Segment Anything Model called SAM, it's a project from Meta that was published in april 2023, which consists of a computer vision model trained from a huge database for image segmentation. The model has been designed and trained to be fast, and its database covers more than 11 million images, making it adaptable to a wide range of tasks. This model was released to encourage the investigation of basic models for artificial research. SAM can also take input indications from other systems, which makes it a versatile model to be coupled and complemented in the development of new applications.
With this model it is possible to request multiple points to work on the image and segment a part of it to convert it into a mask, these are used to show the location of specific objects in an image, the masks help to isolate the parts of the object that you want to segment taking as a reference texture, opacity, shape and color; This model predicts binary masks that verify the presence of the elements of interest, and show as a result the same image with sheets and traces of segmented objects respectively as shown in the images.
However, while SAM works well overall, it does have it's limitations. It can miss fine structures, it can miss small objects, and it doesn't produce very sharp outlines; this is because SAM is designed for generality and breadth of use rather than offering highly interactive segmentation. Also, SAM can process indications in real time, however, the overall performance of SAM is not real time when using a heavy image encoder. Research is still being done to achieve the "text to mask" tagging feature, as it is something that has not been implemented yet.
Fig 1. Picture to be segmented
Fig 2. Segmented picture with the SAM demo
Meta Research developed a demo to interact on the web and learn more about this new model. Next I will demonstrate how this model can be used by accessing the free online demo Segment Anything Model you can explore the functions and new features.
Once you enter it is necessary to accept the terms and conditions, which specify that this demo is not for commercial purposes, they will explain that the images available for testing will only be for testing the model and any modifications will be removed at the end; in addition to the fact that in the event that the user uploads an image, it must not infringe intellectual property, nor the Facebook community standards
By accepting the conditions of use there will be a catalog of images that by selecting one will allow you to explore the model, however, there is an option that says "upload an image", which will allow you to upload your own image that is stored locally and explore the model in this way.
Once an image is chosen, this menu will appear, which will provide options to manipulate the image with the model according to what is desired.
Fig 3. Web Demo Tools Menu
By selecting "Hover and click" and clicking on a part of the image you can choose to segment part of it, this allows us to focus on the specific objects that we want to segment.
Fig 4. Segmentation with tool "hover and click"
By selecting Cut-Outs you can separate the part of the selected image and segment it; that will create a new menu with these images and they can be saved locally on our computer.
Fig 5. Cut segment
There is also the Box option, which consists of selecting with the mouse the specific area of the object that you want to segment. The object is selected until it is highlighted.
Fig 6. Object selected with the "box" tool
And later you have the option of being cropped, and the selected object can be segmented from the rest of the image.
Fig 7. Selected object
Choosing the ✨Everything✨ option will take some time for the image to be analyzed and will show all the possible segments it can give, and selecting Cut out all objects will show all the objects that it was possible to segment by the model.
Fig 8. Image processing with the Everything tool
Fig 9. Segmented objects using the Everything tool
If you want to further explore the features of the demo, you can go in and try your own images to target specific elements; This tool can also be implemented locally to be implemented in your own projects, however, it requires certain requirements such as Nvidia graphics cards and libraries such as pytorch, open-cv and supervision.
- SAM is a tool that allows you to separate objects found in an image, but it is not trained to identify these objects by means of labels.
- This tool has segmentation limitations with small objects or those that are not well identified in the base image. and its delimitations are not usually clear
- One of the advantages of this tool is that it can be coupled with other technologies, in order to develop more complex applications that involve both segmentation and image detection and classification.
- Since SAM is targeted for basic models, there are other technologies that could be used instead.
- Image segmentation models are useful to implement in surveillance, medicine, augmented reality, and robotics applications, among others.
- Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollar, P., & Girshick, R. (2023). Segment Anything. https://arxiv.org/pdf/2304.02643.pdf
- Facebook AI. (2023). Segment Anything. https://ai.facebook.com/research/publications/segment-anything/
- Segment Anything. (2023). https://segment-anything.com/
- DotCSV [DotCSV]. Abril 2023. "SAM: La IA de Meta que LO SEGMENTA TODO!". https://www.youtube.com/watch?v=NmwC393cg_o&ab_channel=DotCSV
This article was developed by:
Software Developer at