The content recommendation algorithm that powers the online short video platform TikTok has once again come under the spotlight after the U.S. ordered its Chinese owner, ByteDance, to sell the app’s U.S. assets or face a nationwide ban.
Here is how it works and why it has attracted more discussion than technology used by its rivals such as Meta’s Instagram, Google’s YouTube and Snapchat:
Algorithms
The algorithms are deemed core to ByteDance’s overall operations, and ByteDance would rather shut down the app than sell it, Reuters reported citing sources.
China made changes to its export laws in 2020 that give it approval rights over any export of algorithms and source codes, adding a layer of complexity to any effort to sell the app.
Academics and former company staff said that it is not just the algorithms, but also how it works with the short video format, that has made TikTok so successful globally.
It’s the app too
Before the emergence of TikTok, many had believed that technology connecting a user’s social connections were the secret sauce to a successful social media app, given the popularity of Meta’s Facebook and Instagram.
But TikTok showed that an algorithm, driven by the understanding of a user’s interest, could be more powerful. Rather than building their algorithm on “social graph” like Meta has, TikTok executives including CEO Shou Zi Chew have said that their algorithm is based on “interest signals”.
While rivals have similar interest-based algorithms, TikTok is able to turbocharge the algorithm’s effectiveness with the short video format, said Catalina Goanta, an associate professor at Utrecht University.
“Their recommender system is very common. But what really distinguishes TikTok as an app is the design and the content,” she said.
The short video format enables TikTok’s algorithm to become much more dynamic and even capable of even tracking changes in users’ preferences and interests across time, going as granular as what a user may like during a certain period of time during the day.
Rapid data collection
In addition, the short video format allows TikTok to learn about user preferences at a much faster rate, said Jason Fung, former head of TikTok’s gaming unit.
“Because it’s in bite size format, it is short video, you’re able to collect data about a user’s preference a lot faster than YouTube, where maybe the average video is just less than 10 minutes long,” he said, “Imagine you’re collecting data about a user on average every 10 minutes versus every couple seconds.”
And the positioning of TikTok as an app built for mobile devices from the beginning also gave it an advantage over rival platforms that had to adapt their interfaces from computer screens.
TikTok’s early entry into the short video market also gave the company a big early-mover advantage. Instagram did not launch Reels until 2020 while YouTube launched Shorts in 2021, both of which lag TikTok in years of data and product development experience.
Allows exploration
TikTok also regularly recommends content that falls outside of users’ interest, which the company’s management has repeatedly said is essential to TikTok’s user experience.
A study, which researchers from the U.S. and Germany published last month, found TikTok’s algorithm “exploits the user interests in 30% to 50% of the recommendation videos” after examining data from 347 TikTok users and five automated bots.
“This finding indicates that the TikTok algorithm opts to recommend a large number of explore videos in an attempt to either infer better the user interests or maximise user retention by recommending many videos that are outside of the user’s (known) interests,” the researchers wrote in the paper named “TikTok and the Art of Personalization”.
Mobilises users into groups
Ari Lightman, a professor at Carnegie Mellon University, said that another effective tactic TikTok has employed is to encourage its users to form groups publicly via hashtags.
By encouraging users to form public groups, TikTok can more effectively learn about its users’ behavior, interest, alignment and ideology, he said.
If TikTok ends up getting banned in the U.S., Lightman said that while the U.S. tech giants certainly have the capability to replicate TikTok with their own products, replicating the user culture enabled by TikTok might be the bigger task.
Chinese advantage
TikTok’s recommendation algorithm was also in large part taken from its Chinese sister app Douyin which was released in 2016. Although ByteDance often stresses that TikTok and Douyin are separate apps, one source with direct knowledge of the matter said the two algorithms remain similar to this day.
In turn, Douyin’s AI was supercharged by the company’s ability to leverage low labour costs in China that saw it hire many content annotators to painstakingly tag all the content and users on the platform.
“Around 2018 and 2019, Douyin worked on having tags on every user. So they would tag every video clip manually. Then they would tag their users based on the video that they have watched,” said Yikai Li, a manager at ad agency Nativex and a former director at ByteDance. “Then they also applied this tactic on TikTok.”
While hiring annotators to tag data is now a common and an important practice for AI companies, ByteDance was early in adopting this strategy.
“It’s a lot of work sorting out these tags. It’s very laborious,” he said, “So Chinese companies have an advantage here. You can afford a lot more people. The cost is cheaper than it is for North American companies.”
—Josh Ye, Reuters