table of Contents
· Introduction
· General description
∘ Goal
∘ Why semi-automatic?
∘ Entering Label Studio
∘ 1 interface + 2 servers
· Implementation (local)
∘ 1. Install git and docker and download the backend code
∘ 2. Configure the interface to obtain access token
∘ 3. Set up backend containers
∘ 4. Connect containers
∘ 5. Happy tagging!
· GCP implementation
∘ 1. Select project/Create new project and set up billing account
∘ 2. Create a VM instance
∘ 3. Configure the VM environment
∘ 4. Follow the previous section and set everything to VM
· GCS Integration
∘ 1. Set up GCS buckets
∘ 2. Create and configure the service account key
∘ 3. Rebuild backend containers
∘ 4. SDK loads images from source repository
∘ 5. Configure destination storage
· Recognition
· References
Creating training data for image segmentation tasks remains a challenge for individuals and small teams. And if you're a research student like me, finding a cost-effective way is especially important. In this post, I will talk about a solution I used in my capstone project where a team of 9 people successfully tagged over 400 images in a week.
Thanks to the Milan Polytechnic. Gianfranco Ferré Research Center, we obtained thousands of fashion show images from Gianfranco Ferré's archive database. To explore, manage, enrich and analyze the database, I used Image segmentation for smarter cataloging and detailed research. Image segmentation of parade photographs also lays the foundation for creating informative textual descriptions for better text-to-image generative ai and search engine approaches. Therefore, this blog will detail:
- how to create your own backend with label studioin addition to the existing segment anything backendfor semi-automatic image segmentation labeling,
- how to host on Google Cloud Platform for group collaboration, and
- How to use Google Cloud Storage buckets for data versioning.
The code for this post can be found in this GitHub repository.
Goal
Segment and identify the names and typologies of fashion garments in fashion show images, as shown in the first image.
Why semi-automatic?
Wouldn't it be nice if a trained segmentation model could perfectly recognize each garment in the show images? Unfortunately, there are none. There are trained models tailored to fashion or clothing images, but nothing can perfectly match our data set. Every fashion designer has their own style and preferences for certain clothes, as well as their color and texture, so even if a segmentation model can be 60% accurate, we consider it a win. So we still need humans involved to fix what the segmentation model did wrong.
Entering Label Studio
Label Studio provides a free, customizable, open source community version for various types of data labeling. One can create their own backend, so I can connect the Label Studio frontend to the trained segmentation model backend (mentioned above) for the taggers to further improve the automatic predictions. In addition, Label Studio already has an interface somewhat similar to Photoshop and a series of segmentation tools that can come in handy:
- Brush and eraser
- Magic wand for selection of pixels of similar colors
- Segment anything backend that harnesses the power of Meta SAM and allows you to recognize the object within a bounding box that you draw.
1 interface + 2 servers
So far, we want to connect 2 backends to the frontend. One backend can do the segmentation prediction and the second can speed up the modification of the taggers if the predictions are incorrect.
Now, let's start the application locally. That is, you will be able to use the app on your laptop or local machine completely free of charge, but you still won't be able to invite your labeling team to collaborate on their laptops. We'll talk about teaming with GCP in the next section.
1. Install git and docker and download the backend code
If you do not have git either stevedore on your laptop or local machine yet, install them. (Note: you can technically skip the git installation step by downloading the zip file from this GitHub repository. If you do, skip the following.)
Then, open your terminal and clone. this repository to the directory you want.
git clone https://github.com/AlisonYao/label-studio-customized-ml-backend.git
If you open the label-studio-customized-ml-backend folder in your code editor, you can see that most of them are adapted from the Label Studio ML Backend repository, but this directory also contains interface template code and SDK code adapted from Tag Studio SDK.
2. Configure the interface to obtain access token
Following the official guidelines of segment anythingdo the following in your terminal:
cd label-studio-customized-ml-backend/label_studio_ml/examples/segment_anything_modeldocker run -it -p 8080:8080 \
-v $(pwd)/mydata:/label-studio/data \
--env LABEL_STUDIO_LOCAL_FILES_SERVING_ENABLED=true \
--env LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT=/label-studio/data/images \
heartexlabs/label-studio:latest
Then open your browser and type http://0.0.0.0:8080/ and you will see the Label Studio interface. Proceed to register with your email address. Now, there is no project yet, so we need to create our first project by clicking Create project. Create a name and description (optional) for your project.
Upload some images locally. (We'll talk about how to use cloud storage later.)
To configure labeling, click Custom template on the left and copy and paste the HTML code from the label-studio-customized-ml-backend/label_studio_frontend/view.html archive. You don't need the four lines of headers if you don't want to display image metadata in the tagging interface. Feel free to modify the code here according to your needs or click Visual to add or remove tags.
Now click Save and your tagging interface should be ready.
At the top right, click the user settings icon and click Account and settings and then you should be able to copy your access token.
3. Set up backend containers
In it label-studio-customized-ml-backend directory, there are many backends thanks to the Label Studio developers. We will use the custom ./segmentation backend for segmentation prediction (container 1) and the ./label_studio_ml/examples/segment_anything_model for faster labeling (container 2). The first will use port 7070 and the second will use port 9090, making it easier to distinguish it from frontend port 8080.
Now, paste your access token to 2 docker-compose.yml files in ./segmentationand ./label_studio_ml/examples/segment_anything_model folders.
environment:
- LABEL_STUDIO_ACCESS_TOKEN=6dca0beafd235521cd9f23d855e223720889f4e1
Open a new terminal and access the cd segment_anything_model directory like you did before. Then, turn on the container of any segment.
cd label-studio-customized-ml-backend/label_studio_ml/examples/segment_anything_modeldocker build . -t sam:latest
docker compose up
Then open another new terminal CD in the segmentation directory and activate the segmentation prediction container.
cd label-studio-customized-ml-backend/segmentationdocker build . -t seg:latest
docker compose up
As of now, we have successfully started all 3 containers and you can check again.
4. Connect containers
Previously, what we did with the access token already helped us connect containers, so we're almost done. Now, go to the interface you started a while ago and click Settings in the upper right corner. Click Machine learning to the left and click Add model.
Make sure to use the URL with port 9090 and enable interactive pre-annotation. Finish adding by clicking Validate and save.
Similarly, do the same with the segmentation prediction backend.
So I like to activate Retrieve predictions when loading a task automatically. This way, every time we refresh the tagging page, the targeting predictions will be activated and loaded automatically.
5. Happy tagging!
Here's a demo of what you should see if you follow the steps above.
If we're not happy with the predictions for, say, the skirt, we can delete the skirt and use purple magic (segment anything) to quickly tag it.
I'm sure you can figure out how to use the brush, eraser, and magic wand on your own!
