Marketplaces like eBay or Vinted are always an interesting target for a variety of actors: from brands willing to monitor the second-hand market to scalpers willing to get something on sale and resell it for profit. Given all this interest in them, these marketplaces needed to protect themselves from bots and fraudulent activity, becoming a hard target to scrape. In this episode of The Lab, our series of in-depth articles, we’ll focus on Vinted, and we’ll see how we can ethically scrape it after studying its backend. What is Vinted? Vinted is a second-hand marketplace born in 2008 in Lithuania, where people can buy and sell used stuff from 21 different countries. According to Similarweb, the website has almost three million visits, while the app has been downloaded 32.8 million times in 2023, with 105 million registered users. Vinted while the app has been downloaded 32.8 million times in 2023 while the app has been downloaded 32.8 million times in 2023 With hundreds of millions of items on sale on the platform, there’s a vibrant ecosystem of sellers and buyers, and, like in every efficient market, people are trying to make some money out of the platform by buying low and selling high, aided by bots in some cases. To mitigate the bot traffic, the platform introduced some anti-bot solutions such as Datadome. In the next chapters, we’ll see how we can scrape public data from the website by using its internal APIs and a technique called “cookie factory”. How the Vinted anti-bot system works Vinted has both a website and an app, so let’s see how they both work to choose the most stable and easiest approach for scraping. The website Let’s say we want the data from a specific category of items on the platform. When browsing the website, we can see that an internal API with the following endpoint handles the pagination of these items. https://www.vinted.it/web/api/core/catalog/items?page=1&per_page=96&time=1745925814&search_text=&catalog_ids=1920&catalog_from=0&brand_ids=&status_ids=&color_ids=&material_ids https://www.vinted.it/web/api/core/catalog/items?page=1&per_page=96&time=1745925814&search_text=&catalog_ids=1920&catalog_from=0&brand_ids=&status_ids=&color_ids=&material_ids Of course, if we try to copy this URL and paste it into a new tab of the browser, we get an error message. This is because we need a set of necessary authorization cookies to see the data. In particular, our focus is on: access_token_web: It’s an authorization token that lasts two hours, as we can see by copying it into an online JWT debugger. refresh_token_web: which is a token that is used as a parameter when we need to refresh the auth token seen before datadome: This is the token that we get when we pass through the security check of Datadome. Its presence unequivocally tells us that the API calls we’re going to make won’t work until we get a valid Datadome token, which can be pretty challenging. access_token_web: It’s an authorization token that lasts two hours, as we can see by copying it into an online JWT debugger. access_token_web: It’s an authorization token that lasts two hours, as we can see by copying it into an online JWT debugger. access_token_web: by copying it into an online JWT debugger. by copying it into an online JWT debugger. refresh_token_web: which is a token that is used as a parameter when we need to refresh the auth token seen before refresh_token_web: which is a token that is used as a parameter when we need to refresh the auth token seen before refresh_token_web: datadome: This is the token that we get when we pass through the security check of Datadome. Its presence unequivocally tells us that the API calls we’re going to make won’t work until we get a valid Datadome token, which can be pretty challenging. datadome: This is the token that we get when we pass through the security check of Datadome. Its presence unequivocally tells us that the API calls we’re going to make won’t work until we get a valid Datadome token, which can be pretty challenging. datadome: The mobile app Inspecting the traffic of the mobile app with HTTP Toolkit gave us another point of view on the backend. The endpoint for extracting products for a certain category is the following https://www.vinted.fr/api/v2/homepage/women?column_count=2&homepage_session_id=f2c89e7b-27b5-4d99-8e76-3e2bf1d55c23&version=3 https://www.vinted.fr/api/v2/homepage/women?column_count=2&homepage_session_id=f2c89e7b-27b5-4d99-8e76-3e2bf1d55c23&version=3 And, in case we need to retrieve more items, we need to add the following URL parameter https://www.vinted.fr/api/v2/homepage/women?column_count=2&homepage_session_id=f2c89e7b-27b5-4d99-8e76-3e2bf1d55c23&next_page_token=feed_max_score:1&version=3 https://www.vinted.fr/api/v2/homepage/women?column_count=2&homepage_session_id=f2c89e7b-27b5-4d99-8e76-3e2bf1d55c23&next_page_token=feed_max_score:1&version=3 As you can see, there’s no traditional concept of page, but the more time we call this endpoint with the same session ID, the more items we get. In this case, the authorization token is used in the request headers We don’t have any refresh_token, and even more interestingly, not a Datadome token. This means that while using the web version, Datadome is configured to protect the endpoint. However, on the mobile app, this is not true. It’s not something unseen before: it happens that mobile endpoints are not under bot protection, mainly for two reasons: They’re usually a bit trickier to find, so unskilled scrapers are not using them, and website owners choose not to protect them Most of the traffic comes from the app, and since these anti-bot systems are billed per request, using them only for the website and not for the app makes the company save a lot of money. They’re usually a bit trickier to find, so unskilled scrapers are not using them, and website owners choose not to protect them They’re usually a bit trickier to find, so unskilled scrapers are not using them, and website owners choose not to protect them Most of the traffic comes from the app, and since these anti-bot systems are billed per request, using them only for the website and not for the app makes the company save a lot of money. Most of the traffic comes from the app, and since these anti-bot systems are billed per request, using them only for the website and not for the app makes the company save a lot of money. From analyzing the website and the app, it seems easier to use app endpoints since Datadome does not protect them, but we still need to figure out how to obtain the authorization tokens. How to get an authorization token This is the real first issue we have to manage for scraping Vinted. In fact, both inspecting the website and the app, there’s no call to any token issuer. It’s not rare to see on some apps a call to a particular endpoint that issues a token that will be used later. In these cases, all you have to do is reproduce the same call, grab the token, and then modify the request headers to include it. In the Vinted case, the token seems to be generated on the flight in the backend: from the first calls, both on mobile and on the website, we see the tokens in the headers or in the cookies (in case of the website). The good thing is that the tokens generated by the website can also be used for the app, and this made me think about a solution for our scraper. The cookie factory The idea is pretty simple: what if I build an API endpoint that, when called, returns me a valid token to use on Vinted mobile endpoint? So I can call it before my script, get the token, and then use the Vinted mobile API endpoint to scrape data. Well, it seems a brilliant idea, but I still haven’t figured out how to generate these tokens, since when I load the home page of Vinted, the cookies are generated in the first place with no API calls. But what if my cookie factory does exactly the same thing? Code example Thanks to Cursor, I’ve created this small API endpoint running locally on my Mac. from fastapi import FastAPI from pydantic import BaseModel from playwright.async_api import async_playwright import asyncio app = FastAPI() class RequestData(BaseModel): url: str cookie_name: str async def get_cookie(url, cookie_name): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context() page = await context.new_page() await page.goto(url) await page.wait_for_load_state('networkidle') cookies = await context.cookies() await browser.close() for cookie in cookies: if cookie['name'] == cookie_name: return cookie['value'] return None @app.post("/get-cookie") async def get_cookie_endpoint(data: RequestData): token = await get_cookie(data.url, data.cookie_name) if token: return {"cookie_value": token} else: return {"error": "Cookie not found"} from fastapi import FastAPI from pydantic import BaseModel from playwright.async_api import async_playwright import asyncio app = FastAPI() class RequestData(BaseModel): url: str cookie_name: str async def get_cookie(url, cookie_name): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context() page = await context.new_page() await page.goto(url) await page.wait_for_load_state('networkidle') cookies = await context.cookies() await browser.close() for cookie in cookies: if cookie['name'] == cookie_name: return cookie['value'] return None @app.post("/get-cookie") async def get_cookie_endpoint(data: RequestData): token = await get_cookie(data.url, data.cookie_name) if token: return {"cookie_value": token} else: return {"error": "Cookie not found"} This simple API allows you to extract a specific cookie from any website using Playwright and FastAPI. It runs a headless Chromium browser, visits the target URL, and searches for the cookie by name. extract a specific cookie from any website We import the following packages: FastAPI: Used to build a simple web API. BaseModel: Pydantic class to validate incoming POST data. playwright.async_api: Asynchronous API for browser automation. asyncio: Required for running asynchronous functions. FastAPI: Used to build a simple web API. FastAPI: Used to build a simple web API. FastAPI BaseModel: Pydantic class to validate incoming POST data. BaseModel: Pydantic class to validate incoming POST data. BaseModel playwright.async_api: Asynchronous API for browser automation. playwright.async_api: Asynchronous API for browser automation. playwright.async_api asyncio: Required for running asynchronous functions. asyncio: Required for running asynchronous functions. asyncio Then we initialize a FastAPI app instance with app = FastAPI(). Next, we define a Pydantic model named RequestData, which expects two pieces of information in the incoming POST request: the URL to visit and the name of the cookie we're interested in. This helps ensure that the API receives structured and valid input from clients. app = FastAPI() RequestData, The core functionality lies in the get_cookie function. This is an asynchronous function that takes in a URL and a cookie name. It uses Playwright to launch a headless Chromium browser, creates a new browser context, and navigates to the given URL. After waiting for the page to finish loading completely, using Playwright's networkidle state to ensure even background JavaScript has settled, it retrieves all the cookies stored in the browser context. get_cookie networkidle The browser is then closed to free up resources. The function then iterates over the collected cookies and returns the value of the one that matches the name provided. If the cookie isn't found, it simply returns None. None Finally, the FastAPI endpoint /get-cookie is defined to handle incoming POST requests. /get-cookie When a request is received, FastAPI automatically parses the incoming JSON and maps it to the RequestData model. The get_cookie function is then called with the parsed URL and cookie_name. If a matching cookie is found, its value is returned as JSON. RequestData get_cookie URL cookie_name You can also find the entire script in the app.py file within The Lab's repository. app.py How to make an API go live Take this paragraph with a grain of salt, since it’s not my area. Based on my research, we can make our API go live by simply using this command: gunicorn app:app -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 gunicorn app:app -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 As far as I understood, Gunicorn, a widely-used Python WSGI server (Web Server Gateway Interface), is what we need for running it. Gunicorn WSGI defines a simple, synchronous interface between your Python code and a web server like Gunicorn or uWSGI. It became the foundation for almost all major Python web apps at the time. A WSGI app works in a very linear way: the server receives a request, hands it off to your app, waits for the response, and only then moves on to the next request. This model is simple and reliable, but it's also limited to one request at a time per worker. It works fine for traditional web apps that render HTML or interact with a database, but it struggles when handling large numbers of concurrent users, long-lived connections, or real-time updates. limited to one request at a time per worker Fast forward to today, and we have ASGI (Asynchronous Server Gateway Interface). It was introduced to bring Python into the modern era of asynchronous programming, enabling things like WebSockets, long polling, and concurrent tasks without spinning up new threads or processes. ASGI asynchronous programming ASGI is what allows FastAPI, Starlette, and newer Django versions to support high-performance async features. Unlike WSGI, ASGI apps can handle multiple requests at the same time within a single worker, making them ideal for APIs, streaming data, and anything that benefits from non-blocking I/O (like scraping with Playwright, for instance). FastAPI Starlette multiple requests at the same time In our case, we're combining Gunicorn with Uvicorn, the ASGI server on which FastAPI is built. This pairing gives us the best of both worlds: Gunicorn for robust process management, and Uvicorn for high-performance asynchronous handling. Uvicorn I didn’t dig the subject further since the endpoint I’m creating is for personal use on my Mac, but consider any security and scalability issues before deploying any similar solution in public. Scraping Vinted, finally After understanding how to get the auth token, scraping data from Vinted is a piece of cake if we follow some tricks. I’m sharing with you the full code inside the file scraper.py, but we can sum up the code in the following steps. scraper.py, Calling the API With the following code, we’re calling the API endpoint and storing the auth token inside a variable in our script. def get_bearer_token(): """ Get bearer token by making a call to the local API """ url = "http://127.0.0.1:8000/get-cookie" headers = { "Content-Type": "application/json" } data = { "url": "https://www.vinted.it", "cookie_name": "access_token_web" } try: response = requests.post(url, headers=headers, json=data) if response.status_code == 200: token_data = response.json() return token_data.get("cookie_value") else: print(f"Error getting token: {response.status_code}") print(response.text) return None except Exception as e: print(f"Exception when getting token: {str(e)}") return None def get_bearer_token(): """ Get bearer token by making a call to the local API """ url = "http://127.0.0.1:8000/get-cookie" headers = { "Content-Type": "application/json" } data = { "url": "https://www.vinted.it", "cookie_name": "access_token_web" } try: response = requests.post(url, headers=headers, json=data) if response.status_code == 200: token_data = response.json() return token_data.get("cookie_value") else: print(f"Error getting token: {response.status_code}") print(response.text) return None except Exception as e: print(f"Exception when getting token: {str(e)}") return None Before scraping the first page Now we can set up the API call to retrieve the first items from the Vinted women's clothing category by creating a random session string and an anonymous user ID, and updating the headers with the new authentication token. if not homepage_session_id: homepage_session_id = str(uuid.uuid4()) if not x_session_id: x_session_id = str(uuid.uuid4()) # Generate a random x-anon-id x_anon_id = str(uuid.uuid4()) url = "https://www.vinted.fr/api/v2/homepage/women" # Query parameters params = { "column_count": 2, "homepage_session_id": homepage_session_id, "version": 3 } # Add next_page_token if provided if next_page_token: params["next_page_token"] = next_page_token # Headers from the curl command headers = { "accept": "*/*", "accept-encoding": "gzip, deflate, br", "accept-language": "it-fr", "authorization": f"Bearer {bearer_token}", "connection": "keep-alive", "host": "www.vinted.fr", "short-bundle-version": "25.17.0", "user-agent": "vinted-ios Vinted/25.17.0 (lt.manodrabuziai.fr; build:30160; iOS 18.4.1) iPhone17,2", "x-app-version": "25.17.0", "x-device-model": "iPhone17,2", "x-device-uuid": "e121b6c28a0ae580f66635c1335acd24", "x-anon-id": x_anon_id, "x-session-id": x_session_id, "x-icloud-identifier": "_e390798f99c3643239475f3ececd12f1" } if not homepage_session_id: homepage_session_id = str(uuid.uuid4()) if not x_session_id: x_session_id = str(uuid.uuid4()) # Generate a random x-anon-id x_anon_id = str(uuid.uuid4()) url = "https://www.vinted.fr/api/v2/homepage/women" # Query parameters params = { "column_count": 2, "homepage_session_id": homepage_session_id, "version": 3 } # Add next_page_token if provided if next_page_token: params["next_page_token"] = next_page_token # Headers from the curl command headers = { "accept": "*/*", "accept-encoding": "gzip, deflate, br", "accept-language": "it-fr", "authorization": f"Bearer {bearer_token}", "connection": "keep-alive", "host": "www.vinted.fr", "short-bundle-version": "25.17.0", "user-agent": "vinted-ios Vinted/25.17.0 (lt.manodrabuziai.fr; build:30160; iOS 18.4.1) iPhone17,2", "x-app-version": "25.17.0", "x-device-model": "iPhone17,2", "x-device-uuid": "e121b6c28a0ae580f66635c1335acd24", "x-anon-id": x_anon_id, "x-session-id": x_session_id, "x-icloud-identifier": "_e390798f99c3643239475f3ececd12f1" } Calling the API and parsing its results. After setting up the headers, we can iterate through the calls on the API endpoint and parse the results. In our case, I’ve set a hard stop after 100 calls or when we’ve run out of products to scrape from a specific category. It’s a simple Python request with some JSON parsing, so there’s no need to comment any further. Conclusions The idea behind this article is to show a different technique for scraping, which uses a cookie farm to enable our programs to bypass API authentication. This can also be used when dealing with anti-bots by generating valid cookies to use in the sessions. In this case, it could be much more difficult, since it would require a stealthier and more human-like browser session to get it. All the code created is for educational purposes only and should not be used to flood Vinted or any other website that works similarly with requests. I hope you enjoyed it, and please let me know in the comments section. The article is part of “The Lab” series by Pierluigi Vinciguerra. Check out his Substack page for more knowledge on Web Scraping. The article is part of “The Lab” series by Pierluigi Vinciguerra. Check out his Substack page for more knowledge on Web Scraping. “The Lab” “The Lab” Pierluigi Vinciguerra Substack
