Summary: This article explores how TikTok can add advanced search features leveraging AI-powered visual recognition.
The Concept
TikTok Visual Search utilizes AI-driven visual recognition, contextual understanding, and content analysis to improve search capabilities. Users can search by uploading images or screenshots to find relevant related content. Users can search for TikTok videos, products, ads, trends and profiles. Multi-modal AI allows the feature to combine data, video content, and text to refine search results based on visual elements.
Ideation & Research
TikTok's Mission
"to inspire creativity and bring joy"
TikTok achieves this by creating an inclusive environment that celebrates various diverse communities.
Current Users
TikTok has around 1.1 billion users across 160 countries. The US-based user age range is 10-19: 32.5%, 20-29: 29.5%, 30-39: 16.4%, 40-49: 13.9%, 50+: 7.1%.
US Audience: About 80 million monthly active users are in the US. Around 60% are female and 40% are male. Around 60% are ages 16-24 and 26% are between ages 25-44.
Gen Z: Around 60%+ of TikTok users are GenZ.
Countries: TikTok is available in about 154 countries with over 75 different languages.
More stats can be found here: TikTok Stats
Market Research & Validation
Some users love TikTok while others prefer to stay away from it. It's important to aim to provide the resources that users want. So, what are some things people still want from TikTok?
Several people agree that TikTok has helped them learn about some cool life hacks, plan travel, find food recipes, and more. While there's the risk of spreading misinformation, several users look up to trusted creators to learn about niche topics.
User Reviews:
"I get tons of food recipes to try out, tons of cool animal facts, a lot of scientific and medical knowledge, ext... You can shape your FYP to give you what you want."
"there's different communities within the platform e.g. gardeners, dancers, gamers and many more use the app and it's not just one thing. It can be informative, you can learn new things, you can get involved in different physical challenges, you can express your own opinions on things, you can be as creative as you like and people there might appreciate you, accept you and support you where you haven't been before"
People enjoy that their FYP's are personalized to their interests. It's a great way for users to gain exposure to random topics and information that they may have never thought of before. Younger users also turn to TikTok search to find information over Google. For example, several people would look for travel creators to look for recommended food spots in the city they are visiting. Video form information is becoming more reliable and utilized by current users. So bringing AI-powered Visual Search will take the search capabilities one step further.
Current Competitors
There are several video creation platforms. TikTok has direct, potential, and substitute competitors. Each competitor brings a different kind of challenge to TikTok, whether that be through specialized niches or leveraging different ecosystems.
Instagram Reels, owned by Meta, is a social media platform centered around photo and video content sharing. Their Reels feature directly completes with TikTok. Reels offer nearly all the tools to create and discover short-form video content, as TikTok does. These tools include editing capabilities, popular audio tracks, filters, and more. Reels are heavily promoted through the explore page similar to TikTok's FYP.
Facebook Reels, owned by Meta, integrated Reels into Facebook after their quick success with Reels on Instagram. It uses its large user base to push short-form video content targeting older demographics.
Youtube Shorts owned by Google, launched shorts to capture the short-form video market. Shorts are less than a minute long and utilize YouTube's large audience and creator base. These videos are promoted on the recommended page alongside traditional YouTube videos.
Snapchat Spotlight focuses on user-generated short-form videos. Spotlights utilizes well-developed algorithms to surface popular and engaging content. These are personalized to the user's interests. However, Snapchat remains popular for its unique messaging forms and AR filters.
Triller primarily targets the music industry through music-driven videos. It attracts large creators to keep its emphasis on entertainment and music-related content. It is popular for its AI-powered video editing features.
Clash (Byte) is a short-form video app, created by the co-founder of Vine. Vine was one of the original ideas of short-form content creation. It fosters a creator-centric community offering several monetization approaches. It's unique for its tight-knit user base and creator-first approach.
Likee is created by BIGO Technology. It offers similar features to TikTok such as filters, music, and video creation. Likee is getting increasingly popular in Southeast Asia with a growing emphasis on live streaming and real-time effects.
Dubsmash Acquired by Reddit, Dubshamsh offers short-form video creation tools with a community of diverse creators.
Kwai created by Kuaishou Technology, Kwai is the largest short-form video creation app in China and Latin America. It includes features such as live streaming, informative videos, and e-commerce.
Key Features Ideas
Image & Clip Uploads: TikTok users can upload images or short clips from their gallery or capture them within the app.
AI-Powered Recognition: Utilize multi-modal AI models to analyze the visual output and output-related content, across the platform.
Related Content Suggestions: Apart from outputting exact videos or link matches, TikTok will also suggest related content such as similar product links, top creators, hashtags, and more.
Instant Shopping Links: Users can find direct links to products they upload. This can be partnered with the TikTok shop, gaining more engagements for TikTok shop and other brands and retailers.
Use Cases
Here is a list of a few key use cases but not limited to:
Shopping & Product Identification: TikTok users can scan or upload images/videos of products to find TikTok-suggested reviews, tutorials, ads, or links to find the product.
Food Recipes: Upload an image of a dish and find relevant TikTok recipes, cooking tips, grocery lists, and restaurant recommendations.
Health & Wellness: Upload images of wellness products or fitness content and find related reviews, fitness tips, tutorials, and more.
Fashion & Style Discovery: Upload a photo of an outfit or clothing item and find similar outfits, fashion creators, style tips, and shopping links to similar brands.
Travel Planning: Upload an image of a place or event or simply add your locations and find travel content, restaurant recommendations, activity suggestions, etc near your area.
Planning and Design (PRD)
Objectives
The goal of TikTok Optic Search is to improve TikTok's search experience by allowing users to utilize images to find related TikTok content. This AI-powered tool leverages deep content analysis, visual recognition, and contextual understanding to deliver highly relevant content.
User Personas & Stories
As a consumer, I want to search by scanning or uploading a product I want to learn more about through TikTok reviews and buying options.
As a traveler, I want to upload a photo of a landmark and learn more about nearby events or restaurants.
As a food lover, I want to upload a picture of a dish to learn about nutrition facts, recipes, and related cooking tips.
As a stylist, I want to upload a picture of an outfit and find similar clothing items or shopping links.
Features Scope
Image & Clip Uploads
Users can upload images and videos from the app or their gallery
Utilization of computer vision models helps extract key features
AI-Powered Recognition
Combines image recognition and NLP for improved search accuracy
Models are trained to provide results based on both visual data and text input
Contextual Content Matching
identification of relevant TikToks by matching visual elements to current video data
utilizing video metadata and current trends increases contextual accuracy
Instant Shopping Links
Links to e-commerce partners or TikTok shop
Monetization option for creator's affiliate products by posting direct shopping links
Search Result Categories
Group results in categories, such as:
Similar Visual Content: TikTok videos that contain similar visual elements like items and objects
Shopping: Links to buy the searched product
Tutorials: Videos offering guides, how to's, and more
Reviews: Videos providing reviews on a given item or topic
Trending: Content based on emerging trends relevant to the search
Personalized Search Results
Search results are customized based on the user's history, interaction, and preferences
adaptable algorithm based on feedback loops for more accurate and personalized suggestions
Seamless User Interface
intuitive UI that allows users to switch between different search categories
Integrated AR tools to overlay information for a given object
Functional Requirements
Image Processing: Users can upload images directly from their device or through the in-app camera. The images are processed in real-time using AI models to identify patterns or objects.
AI-powered Content Matching: Analyze and classify image data. NLP models analyze related metadata and textual context from highly related TikToks.
Search Results Display: Results will be displayed similarly to TikTok's current scrollable feed with tabs for each category like shopping links, reviews, etc.
Content Personalization & Feedback Loop: Integrate recommendation engines based on users' interaction history. User feedback can be utilized to refine search results.
E-commerce & Monetization: Integrate TikTok Shop for in-app purchases. It can also display related affiliate links and product tags.
Non-Functional Requirements
Security & Privacy: Ensure user input is processed securely and meets compliance requirements with GDPR and CCPA for data protection.
Performance: Real-time image processing and optimizing AI models for low latency.
Cross-Platform Compatibility: The feature should be functional for both, iOS and Android versions.
Scalability: The feature should be able to support a high volume of concurrent users/searches.
Technical Considerations
Architecture Overview
Frontend: Mobile application (iOS & Andriod)
Backend: Architecture to handle image processing, AI model inference, search query handling, and recommendation algorithms
AI/ML Pipeline: Image recognition, contextual analysis, and content recommendations
Data Infrastructure: Real-time data collection, data training, and integrations with TikTok's current infrastructure
Core Technical Components
Image Processing & Recognition
Computer Vision models such as ResNet can be used for object detection, classification, and feature extraction.
Multi-modal models can extract both visual and contextual features of a given image.
Utilization of public cloud features to manage data storage and complex analysis.
Search Queries & Content Matching
Natural Language Processing (NLP) models help analyze text content associated with images and videos.
A combination of visual, text, and audio features helps match images to existing TikTok content.
Personalization
Utilize user data, such as their history of likes, comments, follows, etc to personalize search results.
Implement reinforcement learning to fine-tune search results for each user.
Ranking algorithms can help prioritize content based on user preferences, trends, and relevance.
UI/UX
Display search results based on categories such as reviews, tutorials, etc.
Intuitive UI where users can easily switch search results, apply filters, alter search, etc.
Supports several screen sizes, and device types.
AI/ML Model Development
Model Training & Fine Tuning
Train computer vision models utilizing image datasets and TikTok's data sets.
Fine-tune models using TikTok's data to convert general image recognition models to platform-specific
Transfer learning techniques to pre-trained models to create more personalized responses to several use cases
Data Display
Implementing low-latency infrastructure for real-time model inference
Using containerization and orchestration tools helps with scalable deployments and ensures reliability for peak usage
Edge computing to offload certain tasks to the user's device helps where real-time latency is critical
Data Infrastructure and Pipelines
Data Collection & Labeling
Store user interaction data to continuously improve model accuracy
Develop automated & manual systems of annotating content to improve search relevance.
Update and retrain models based on feedback loops
Real-Time Data Processing
Implement data pipelines to handle real-time streaming data from user interaction
Hybrid approaches allow for online processing to handle real-time queries and batch processing to update the recommendation system
Data Privacy & Security
Image Data Handling: Apply security measures and compliance structures to ensure the security of image data storage and processing
Anonymizing user data protects the user's information while still helping models learn new patterns
Give users the option to opt-in when collecting data to improve the model
System Performance and Scalability
Scalability Considerations
Horizontal scaling allows for more instances to run simultaneously during peak load
Caching strategies help reduce the load on core services
Performance Optimization
Techniques like quantization, pruning, and parallel processing drive low-latency model inference
Load balancing distributes requests evenly across several services to keep response times consistent
A/B Testing several UI configurations to optimize user experience
Development, Monitoring, and Maintenance
CI/CD Pipelines
Implement automated testing for performance, functionality, and security checks
Canary deployment strategies ensure that new updates are rolled out with minimal disruption
Monitoring and Analytics
Real-time monitoring tools help track system performance, user interaction, and model accuracy metrics
Analyze search engagement data helps improve the algorithms and feature improvements
Model Retraining and Updates
Implement periodic model retraining using new data, trends, and patterns
Implement learning techniques using real-time data ensuring that models adapt to preferences.
Wireframes & Prototypes
Sample Low-Fidelity Wireframes
Sample High-Fidelity Diagrams Wireframes
Success Metrics
Search engagement rate, search relevance score, click-through rates, and shopping conversion rates all measure the feature's success. Users interacting with the search results and leaving impressions showcase users' engagement rates. Users who find the results useful to them or those who make purchases based on shopping suggestions help refine the overall relevancy score.
Risks and Mitigations
Several risks can come with implementing a new feature. Here are a few to consider:
There could be cases with inaccurate image recognition leading to irrelevant search results. However, continuously training the models and implementing user feedback systems can help refine the model and results
Users could feel overwhelmed with complex UI/UX for searching. Working with the design team to develop a simple UI with categories and clear navigation between search and results.
Using image data can cause some privacy concerns for users. The feature could ensure clear communication about data usage and even provide an option to not retain images uploaded during the search.
Project Roadmap
Phase 1: Research, Ideation, and Validation
Market Research (Week 1-2)
Study competitor's visual search capabilities such as Pinterest Lens or Google Lens
Identify gaps and opportunities for TikTok
User Research (Week 3-4)
Create surveys and focus groups to document user needs and pain points
Create user personas and stories specific to Visual Search Capabilities
Conceptualization (Week 3-4)
Brainstorm design and feature ideas
Create wireframes and UI/UX flow diagrams
Prioritize feature ideas based on user needs and current tech trends
Technical Planning (Week 5-6)
- Coordinate with engineering teams to define tech stack, architecture, and key integrations
Prototype Development (Week 6-8)
Develop low-fidelity prototypes to validate core feature functionality
Document initial user feedback and re-evaluate priorities
Phase 2: Prototyping & User Testing
High-Fidelity Prototype Development (Week 9-11)
Develop detailed UI/UX Designs
Develop high-fidelity prototypes with key functionalities
Advanced User Testing & Feedback (Week 11-12)
Conduct extensive user testing using the high-fidelity prototype
Document and analyze the feedback to make feature improvements
Model Selection & Training (Week 13-14)
- Based on the technical planning phase, select computer visions and NLP models, and set up data pipelines for model training
Integrate Frontend & Backend Systems (Week 15-16)
Integrate backend services with frontend prototype
Implement APIs for image processing and search results
Phase 3: Model Development & Integration
Full-Scale Model Training (Week 17-20)
Train models based on TikTok-specific data
Improve accuracy through fine-tuning and transfer learning techniques
Backend Development & Infrastructure Setup (Week 21-24)
Deploy backend services for model inference and data handling
Implement caching and load balancing for scalability
Finalization of UI (Week 21-24)
- Implement dynamic UI features and finalize all design choices based on user feedback
Personalized Recommendation Algorithm (Week 25-28)
- Provide personalized results based on a customized recommendation engine
Phase 4: Beta Testing & Iteration
Internal Beta Launch (Week 29-30)
- Launch the feature internally and monitor the performance, identify bugs, and collect feedback
Bug Fixes (Week 31-32)
- address performance issues or bug fixes documented from the internal beta launch
Beta Testing with selected users (Week 33-34)
conduct beta testing for a larger audience or a select group of users
document their feedback based on functionality, usability, and satisfaction
Final Iterations & Improvements (Week 35-36)
- implement any final feature improvements or UI refinements and conduct additional user testing
Phase 5: Full Launch
Production Deployment (Week 37-38)
- Deploy the Visual Search feature to all TikTok users with minimal disruption
User Education & Onboarding (Week 39-40)
- create content or onboarding flows to introduce and educate users on the new feature
Marketing Campaign Launch (Week 41-44)
- Work with the marketing team to develop marketing campaigns to showcase real-like use cases
Post-Launch Monitoring & Support (Week 45-48)
continuously monitor the performance, engagement, and feedback post-launch
provide support and release updates to address any issues
continuously analyze the engagement data and decide to kill or enhance the feature
Stakeholders
Product Management: Define features and develop go-to-market strategy
Engineering: responsible for determining architecture, development testing, and release
UX/UI Design: Design the user interface and experience for Visual Search
Sales & Marketing: Develop creative and strategic campaigns to promote this new feature
Development and Testing
Agile Development
Break down the development process into sprints to focus on building the core functionalities. Determine which project-tracking software works best for the team. Hold standups and sprint retrospectives to track progress and address any issues.
Here is an example of a sprint breakdown:
Sprint Overview:
Goal: The goal is to develop shared whiteboard features
Duration: 2 weeks
Members: Product Manager, Frontend Engineer, Backend Engineer, Teach Lead, UX/UI Designer, QA Tester
Epic 1: Image Processing & Recognition
Assignees: Frontend Devleopers & Backend Devleopers
User Story: As a user, I want to be able to upload images to initiate search.
Task 1: UI component for image upload
Task 2: Backend development to handle image upload
Task 3: Store uploaded images temporarily in the cloud
Iterative Testing
Focus on user experience, performance, and accessibility for internal and external testing. Gather feedback from usability testing from several users and make continuous improvements.
Quality Assurance
Conduct QA testing, including functional testing, security testing, and stress testing to ensure the feature can handle multiple users.
Beta Launch & Feedback
Select a target user group to launch the feature and get early feedback. Continuously measure performance against KPIs such as user satisfaction, engagement rates, and search relevance. Document these insights and make necessary improvements to the functionality, experience, and performance of the product. Refine AI models using data from feedback loops and user behavior.
Full Launch & Marketing
As part of the PRD, plan a full rollout to all TikTok users. Ensure the platform is fully functional and all bugs from the beta tests are resolved. Work with the marketing team to develop a marketing campaign highlighting the value of visual search and focusing on trend-based communities. Work on a GTM (go-to-market) strategy where the company could collaborate with popular creators to showcase the feature through personalized content such as tutorials or challenges, etc.
Post-Launch Monitoring & Maintenance
Product managers can use a couple of frameworks to monitor feature metrics. Two popular ones are HEART and AARRR, which track growth, engagement, retention, user satisfaction, and revenue. Using data analytics to gain insights on engagement and retention can help identify high-value improvements, expand feature capabilities, scale partnerships with brands, and continuously provide relevant search results. Collecting user feedback helps improve the feature or introduce new enhancements to increase user satisfaction. Finally, based on revenue and user satisfaction, decide if planning new enhancements for the product will be a good future investment (the final stage of the product lifecycle).