Interview: Flipkart’s CPTO on building experiences for its next-generation of users

How has Flipkart invested in regional languages? What are the challenges when it comes to creating a platform around these? What are the tech investments Flipkart has done around these?

Two years back, we decided to strongly focus on vernacular languages and driving adoption for these. We did an ethnographic study to get some insights on user pain points, what were the barriers for them.

Based on that deep user research work, we came up with some operating and design principles. So today, we have 11 regional language interfaces on our app. This covers more than 80 per cent of the native language speaker base in the country.

Jeyandran Venugopal, Chief Product and Technology Officer at Flipkart. (Image credit: Flipkart)

We realised that it’s not enough to just change the app scaffolding itself, all the navigation, help screens and others to the vernacular language. We wanted to make it a deeply immersive experience for such users and rely on the right judicial mix of translation versus transliteration. We went around translating/transliterating several millions of words into native scripts. And also did several experiments to fine-tune the user experience (UX), user interface (UI) of the platform.

As of the end of 2020, the percentage of daily active users that we used to see on the platform of users, who would rely on a vernacular version of Flipkart, was somewhere around 10-12 per cent. It now stands at 18 per cent, which is significant growth. We also see a significant level of adoption from tier-two to tier-three cities.

Further, for new users that come on our platform, especially from tier-two and three cities, the percentage of users adopting vernacular languages is very high. In some geographies, as high as 70 per cent plus of all new users prefer to use a vernacular language version of our platform.

Nearly 95 per cent of people who choose the vernacular version of the platform do not switch back to English as the default option. So, that gives us confidence that the interface is working, it is able to give them value and it is engaging them.

We are using artificial intelligence and machine learning and going beyond just translation. We have invested in natural language processing (NLP) and understanding technology, where we are trying to get the entire user comments in a regional language. Because these play an important role in decision-making for users. And most of those comments are in English right now. Right now, we have converted these for Hindi. We will continue to expand it for other languages as well as invest more in building our NLP capabilities.

What about voice search experiences and the challenges around building these? What are the trends that you’ve seen given that many users in India are not comfortable with typing?

We have seen increasing adoption of Voice Search, especially from the new wave of users that are adopting the internet. Voice Search today is available across Flipkart in both Hindi and English. In fact, even mixed mode speech is supported. We are going to continue to invest in expanding that to all the regional languages that we support over a period of time.

The Flipkart voice search. feature. (Image credit: Flipkart)

We know that typing in the Devanagari script is extremely difficult and that came through our user research as well. Automatic Speech Recognition (ASR) is a technology that is the linchpin of making Voice Search work. It is a combination of ASR and NLP and understanding the language itself.

Further, once you have decoded the speech into textual input, how do you convert that from textual input to a semantically understandable phrase? Many times, you may also want to invest in text-to-speech capabilities so that you are able to respond back to the user via speech.

We did a product called Discovery Assistant where we also had a text-to-speech component where the assistant will also answer the user back in speech.

In terms of Voice Search, understanding of e-commerce phrases and intent, we do think that this area needs substantially more investment over a long period of time for the entire industry. It is only going to get better, and it is going to drive even more adoption over a period of time.

We already do like a few million daily queries on an average day around Voice Search. Tier-three plus cities constitute more than half of the overall voice queries.

How is Flipkart using data sciences to enhance the consumer and seller experience?

Data science is pervasive across every area of our business. Take the example of personalising the experience for any user. We have a catalogue of millions of products and customers want to see a version of the page that feels customised to them and one which surfaces the right kind of deals. So how do we do that?

Products are seen in a Flipkart warehouse. (Image credit: Flipkart)

We have a highly-personalised homepage. The deals, all the widgets, the banners are optimised to cater to a user’s personal interests which we learn over a period of time. This is based on view history, click history. You may or may not have bought anything, but even what you search for and then preferentially click one product versus something else gives us some signal about your preferences. We use that to build a very deep personal profile for every user and algorithmically create the right experience for them.

We also use data science quite extensively for getting the product to our customer’s doorstep. For instance, in India, there is no standardised grid system for listing addresses. Many times, addresses are extremely non-standard. So how does our delivery agent then deliver a certain number of packets within a certain time period?

We have developed geolocation intelligence platforms, which do geocoding and reverse geocoding to identify the right geocode or any natural language address. This is also based on the feedback that we get from our people on the ground. We also do a lot of optimisation work on the warehousing side, route optimisation algorithm for the planning. All of that uses elements of data science.

From a seller perspective, we have deeply invested in computer vision technologies to help them create catalogue listings for our users. Because not all of our sellers will be able to create the highest quality catalogue pictures. So we use computer vision to be able to automatically infer some of these attributes and fill this on behalf of the seller. All of this happens behind the scenes. This is already live for several verticals. We are continuing to expand coverage for it.

What about supply chain issues and challenges during Covid-19 lockdowns? What were the key learnings?

So when lockdowns happened and when zones were cordoned off or we had to restrict ourselves to doing all the essential goods only, it did throw off our overall supply chain. And the supply chain is our continuously running machine.

Now, we have developed a lot of systems over these last two years to deal with these things. Earlier, we did not even have systems that could understand non-serviceable areas within the service zone. How do you even represent that and make sure that customers don’t get disappointed? In earlier lockdowns, even the definition of what was allowed, what was not also kept changing.

We now have a more robust system that is able to handle all of these disruptions without the entire supply chain going into disarray. Earlier, we had goods piling up in sortation centres, and these are not equipped to hold these products. But when these kinds of things happened earlier, we ended up discovering some of these failures in our design. We have invested in technology to handle these disruptions in better ways.

Flipkart Camera feature works with categories such as furniture. (Image credit: Flipkart)

What about Augmented Reality and e-commerce? You also announced a partnership with Snap last year. Can you elaborate on this?

Customers are looking for an experience where there is no compromise. And in an offline setting, when you’re buying a consumer durable, you can physically see the product and assess it in your mind. Those kinds of things have always been the issue with e-commerce in the past because you’re not physically able to see the product.

Now, with some of these technologies such as AR that are able to bridge that gap, we can even do one better than offline commerce. With AR, you are intermingling digital reality and physical reality completely in a seamless way. We can actually project the digital object onto our physical reality and be able to see how it looks in our surroundings before we take the call to buy it.

For example, this will be true for furniture. You could even see how well a colour matches your surroundings. And similarly, for beauty personal care. All of these use cases are very fairly well known.

We have partnered with Snap as it has the largest ecosystem of creators for AR functionality. We want to enable our customers to benefit from the creativity of this ecosystem. Where it makes sense, we believe that the AR experience will make the buying experience better for the user. For instance, you could be buying watches. Now with an AR filter, you can just slap the watch on your hand and actually see how it looks. There are lots of interesting opportunities that we have in mind and want to build for this partnership.

We have already introduced our own Flipkart camera in July 2021 for a few categories such as furniture, luggage, beauty categories, large appliances. And, we were able to see significant uptake and conversion for users, when they had this experience versus otherwise in our internal controlled testing. That’s when we realised we had to scale this up and started looking for partners to leverage the ecosystem.

Snap was kind of an automatic choice that we decided to work with. Our own experience has launched, but not at scale. Now we will rapidly scale it.