Finding similar photos on your computer - AntiDupl NET

A little about search

When we talk about search, we immediately imagine the Google search engine with a form for entering a text string and many hundreds of results of links to the pages found. However, let's think about the subject of our search.

What are we looking for?

I named the three largest search engines and they all allow you to search for videos by text and images.

Disadvantages of modern search engines

Unfortunately, they all suffer from the following problems:

• The exact title of the found video is not always available. Instead, the actor may be correctly identified and other images of him or her may be presented.
• There is no exact indication of the position in the video. And this is very useful information.

Yes it is. Try it yourself and you will see that I am right. Search engines are inherently vague. Look at the screenshot above, the one showing Tom Hanks. There is neither the title of the film nor the position in which it was made.

Search by video excerpt on forums

The human factor may be inferior in some ways to machines in searching for information, but it definitely wins in terms of quality. You should not discard sources where other users can help you. People often help each other on thematic forums and online services. Try registering and leaving your question in the forum.

Forum for searching videos by excerpt

Ask for help on other resources, for example, on the Mail.ru Answers or Yandex.Knowledgers service.

On “Answers” ​​people very often answer questions, for some it is beneficial, for others they are simply interested in helping. You can upload part of the video you are looking for on the server so that others can identify it and answer your question.

Home page Answers Mail.ru

If this help is very important for you, you can place your question in an advertising banner on “Answers” ​​for a fee and receive 10 times more advice. It is not necessary to look for forums and services with the topic of video search. Every popular resource has a section for questions or flood. Use them to achieve your goal.

Formulation of the problem

Before we begin solving the problem, let's try to describe it. So what do we want?

Desired query execution speed

Nowadays, no one will wait several minutes for a search query to be completed. However, the volume of data and computation may be such that it will take some time to process the request. We need to compromise. We will conditionally limit the search query execution time to 10 seconds (± a few seconds). This, on the one hand, will allow the browser not to break the connection, and, on the other hand, will give time to the scripts to process the information.

What is the volume of data?

Let's think about it in our heads.

Number of videos

According to the IMDb film database, approximately 2.6 million films have been produced, including individual episodes of TV series, cartoons and short films.
(Information as of November 13, 2020). To begin with, let's limit ourselves to the round number of 1 million videos. It is clear that we are not even trying to touch on YouTube and other similar services, where the volume of videos is many times larger. And most importantly, this snowball will only grow.

Number of frames

Some movies or TV series episodes are quite short.
There are 15-20 minutes. On the other hand, there are quite a few films that last up to 2 hours or more. Without further ado, let’s take the average video length to be 1 hour. A large number of films are shot at 24 frames per second, but there are also faster ones. Nowadays, everyone can make their own film, and the frame rate in it can be 60, 100, 200 FPS or higher. It all depends on the video camera, camera, action camera, smartphone, video surveillance camera, etc. (underline as appropriate). All in our hands. But, let's take, as a first approximation, the frame rate of an average video to be 30 FPS.

In this case, the average video will have:

30 FPS * 3600 sec = 108,000 frames

Rounding up, we get that the average video has about 100,000 frames.

Data volume

What is the amount of information stored about one frame?
Obviously, this value depends on the algorithm for comparing frames in our database with a given sample. We use two algorithms to compare data. One of them requires about 30 bytes per frame, the other about 10 bytes. Let's take the average - 20 bytes. This means that to store information about 1 million videos you need

1,000,000 videos * 100,000 frames * 20 bytes = 2,000,000,000,000 bytes

Simply put, we will need about 2 TB to somehow describe all our frames. Which, generally speaking, is not so bad, because this amount of information can fit on a modern HDD or SSD drive. On the other hand, this information should be somehow organized, otherwise even just reading 2 TB will take a lot of time, and we agreed that the user will not wait more than 10 seconds.

Even if we read information from the disk at a speed of 500 MB/sec, it will take us 2000 seconds, i.e. more than half an hour!

How many servers do we need to search within the specified time?

If we assume that we store information evenly on several servers, then, in this case, the amount of information processed to complete one search query is reduced. For example, if we have 10 servers, each of them will need to process not 2 TB of information, but only 200 GB. Or if we have 100 servers, then we will need to process not 2 TB, but 20 GB of information. In principle, the specified amount should be enough for the functioning of such a search system.

How many requests per second can such a system handle?

It is difficult to answer exactly, but most likely a maximum of several dozen requests per second.

Video Search Page Options

Above we explained how to search for VKontakte videos online, now let’s take a closer look at the section’s filters. Here are the parameters there:

1. Duration. You can select “Long” or “Short”, thereby narrowing your search;
2. Date of. This option allows you to sort entries by download date. The minimum period is the last 24 hours, followed by a week, month and year;
3. Sorting. Here you need to select the relevance of the filter parameters or, what to consider first - the requirements for the duration or date of the download;
4. Quality. Check the box there if you need high-resolution recordings;
5. No limits. If you check this box, the site will sort absolutely everything for you - from short videos shot for fun to content marked 18+.

Let's say you need to find a video of Pavel Volya's speech dedicated to his wife on her birthday. This is a touching piece of the solo program, the total duration of which is about 2 hours. What are we doing?

• In the search bar we write “Pavel Volya for Laysan”;
• In the filters we indicate “Short” (so that you are not offered to watch the entire concert);
• Enter the date “For the year”. Despite the fact that the performance took place more than 2 years ago, someone could easily have uploaded this video again. For effectiveness, we set the longest interval;
• Sort by duration;
• Check the box only in the “No restrictions” option;
• Click "Search";
• The necessary piece of speech in our search results was found immediately on the first page.

This is roughly how the VKontakte video search engine works, as you can see, nothing complicated!

What was done

First, we implemented a search by video fragments. However, image search was soon implemented.

Story

July 1, 2019

On this day, the first version of the VideoColor package was released. It included three parts:

• Manager (indexing source video)
• Server (the server part that accepts requests and looks for a match in the index database)
• Client (client application that allows you to play AVI files and send search requests to the server).

March 2020

A website was created with the ability to identify videos by uploaded video fragment.

April 14, 2020

The first version of the application has been released for identifying videos and determining the position of the video being played using capturing the contents of part of the monitor screen.

June 23, 2020

The first version of the application has been released to add index and descriptive information about videos to the site's database.

Search by video fragments

main idea

Let's consider video as a sequence of images. For each image, find the average value of red, green and blue. We get three graphs depending on time. Let's build and save these graphs for each video we want to index.

Having received a video fragment for identification, we will construct these graphs for it as well. Let's compare the resulting graphs with the existing ones. Of course, comparisons will have to be made across the entire length of each source film. If the difference between the graphs at a specific point is less than a certain value, then we consider that the problem has been solved.

It is worth noting that this is a simplified diagram. There are several points that differ in the working scheme from what is described here. But, in general, this is exactly the idea.

pros

• Relatively small index size. One hour of video in indexed form takes about 1 MB. Thus, 1000 movies, each about 2 hours long, will take up about 2 GB when indexed.
• Fairly accurate search. Even if the video has been compressed several times, if it looks visually satisfactory, then the fragment will most likely be found.
• For the vast majority of search queries, short fragments of 5-10 seconds are sufficient for correct identification.
• The search quality depends slightly on the video resolution (within certain limits).
• The search is carried out exclusively by video. Audio is completely excluded from the process. The upside is that multiple versions of the same movie with different audio tracks will result in the same movie being searched. This eliminates unnecessary duplication and, as a result, saves resources.

Minuses

• The search must be carried out from beginning to end. Those. When a request is received, it must be compared with all the samples in the database. This imposes certain restrictions not only on the type of memory for storing information, but also on the size of this memory itself. In order to get an answer in a few seconds, it is necessary that the indexes be in RAM. The larger the database, the more space in RAM is allocated for storing information and the longer the search will take. For example, for 2-channel access and when using DDR3 memory with a frequency of 1600 MHz, searching a database of 12 GB will require a minimum of about 0.5 seconds. For a 48 GB database it will take at least about 2 seconds.
• For very dark or very light places in the video (usually transition effects between scenes) on short source fragments, the search will not work well. There will be numerous coincidences. Which, in general, is quite understandable, but unpleasant.
• There will also be problems with identification with the opening screens of video production companies or with TV series. Which, in general, is also quite understandable. This is not an algorithm problem - it is duplicate data.
• The quality of the search can greatly depend on how the video is trimmed around the edges.

Image Comparer

Perhaps the best program for finding and removing duplicate images. Image Comparer has a convenient and simple interface in Russian and supports working with all popular graphic formats. The comparison is made based on the image content itself, the so-called digital fingerprint. In this case, the degree of similarity is indicated as a percentage. Found duplicates can be moved, copied or deleted.

Download: https://www.bolidesoft.com

See also >>TinEye-service for searching duplicate images. Working methods for uniqueizing images

See also >>Finding duplicate images. Unique Image Comparer program

Search by Images

main idea

We divide the original image into M x N table cells. Find the average value of the red, green and blue components in each of the areas. Actually, the set of these values ​​will be the characteristic of this image, with the help of which we can distinguish them all from each other. We enter this characteristic into the database along with a pointer to the video description (Video ID) and the sequence number of the frame in the video. The only question that remains is, what values ​​do M and N take? We used 5 x 5, but you can try other values. With small values ​​of these parameters, there is a chance that we will have many duplicates, and with large values, we will waste a lot of memory.

However, that's not all. If you continue to search using all these characteristics, it will take a lot of time to process each request! How to be? You can calculate the average value of the R, G, B components for this image and, based on these values, group them in the data array. For example: R=200, G=188, B=212. In this case, we enter information about the frame into the appropriate section or add a field to the table. And when searching, we similarly define these components and search taking into account these parameters. Thus, we narrow the scope of the compared data many times and speed up the search.

To be honest, this is only in theory, in practice everything is a little different. But this is a topic for a separate article.

pros

• Relatively small data size.
• It is possible to divide all data into groups and search by groups, which significantly speeds up the search.
• Does not require permanent storage of large amounts of data in RAM, unlike the previous method.
• Low probability of error.

Minuses

• Due to the fact that after recoding the video may differ slightly from the original, and JPEG encoding (when searching by image) changes the original and the group may be determined incorrectly. This requires either expanding the range of the group (resulting in a decrease in search speed) or additional search queries (also slowing down the search).

DupliFinder

A simple and convenient program for finding and removing duplicate photos and other images. The search is carried out by comparing MD5 checksums. Otherwise the program is similar to Dup Detector. It supports working with subdirectories, displaying the percentage of similarity, and previewing comparison results. Selected duplicates can be deleted. Disadvantages include somewhat slow speed and the need to install a Java package. Interface language is English.

Download: https://duplifinder.codeplex.com

Tools

To date, several applications have been written, some of them have become outdated and are no longer supported.

Video search (client part)

• Via a web form on the site
• Via the Video Color Capture application

Video search (server part)

• Video Color Server. There are two versions: Windows (runs as a service) and Linux (runs as a regular user, launched via crontab).

Adding a video

• Via the Video Color Creator application

Areas of application for video search by video fragments

• Identification of old and unknown video films.
• Finding and cutting off advertising blocks.
• Checking parts of the video to see if they are borrowed from other films (plagiarism).
• Determining the exact publication date and name of the show (program) if this information is not included in the repost.
• Determining the more or less exact position of the streaming video being played if a previously indexed video is being broadcast.

Identification of old and unknown video films

Let's say you have a file with a clumsy name. The opening title screen is either missing (the author's intention) or cut out. What kind of movie is this? I would like to read the description and comments from those who viewed it.

Finding and cutting off advertising blocks

Example: You have your own custom video player and you want your users to see your own ads when watching streaming video rather than ads from central channels. Checking parts of the video for borrowing from other films (plagiarism) Example: If there is a suspicion that someone is using your video (shot from a quadcopter) in their video.

Determining the exact publication date and name of the show (program) if this information is missing in the repost

Example: You are watching a video show hosted on an unknown site. You may even know what the show is called, but you don't know when it aired. A year ago or two?

Determining the more or less exact position of the playing streaming video if a previously indexed video is broadcast

Example: This may be needed if you want to attach an application to someone else's streaming video that displays captions or other contextual information (maps, links, news, etc.). First, the video is captured, the index is calculated, the video and position on the server are identified, and then the application displays contextual information synchronized with the playing video in a separate window.

Establishing the location of the action on the video clip

An additional feature of video content is the location where it was filmed. In addition to the video participants themselves, the determining factor can be the background details. For reliable identification, it is necessary to determine the location of the shooting. Finding the location of the shooting and the work associated with this process falls into the field of detective investigations. If the video is found on the Internet and the primary source is information from a website, it is necessary to identify the location of the action. At the same time, data on the origin of the content in the context of the author is taken into account, and data regarding its placement is clarified. It is based on various activities to search for “clues” - important information, based on which the required information can be identified. These include both the primary technical parameters of the video stream and small details that are invisible at first glance.

In any data search and identification, to increase the reliability of the result, a necessary condition is the presence of two data sources. Finding the shooting location is one of the basic stages of the entire identification process, which will allow, in addition to visual data, the opportunity to find information from an alternative source. Ultimately, the search for a shooting location can be narrowed down to the city, and later to the exact address. At the same time, other relevant data may appear from third parties who were indirect participants in certain situations, witnesses or have important information. The task of finding a shooting location becomes more difficult in situations where the video of interest was filmed indoors and the situation does not allow obtaining additional information. To search for such opportunities, IT investigations are used, the purpose of which is to study data and publications in general to find the author and his territorial affiliation.

Identifying the person with the video

To reliably identify the person in the video, especially when the low quality of the content does not allow the person to be identified by visual characteristics, it is necessary to search for the participants in the shooting. Using the characteristics of a video clip or fragment as initial data, it is necessary to find the persons who took part in the creation of the video. If there is a high-quality “picture”, this stage may not be necessary, however, it is in this way that the version regarding the visual similarity of a person with a specific person can be confirmed or refuted. In most cases, the situation with the search for a person with a video is a detective investigation, during which measures are taken to identify the video itself, search for the location where it was filmed, and identify a number of minor aspects that make it possible to identify the participants in the process. In the case when the shooting was done intentionally, the author or the location of the camera is known, the task is simplified due to the narrowing of the search circle. First of all, we are talking about a geographical criterion, since when analyzing video from the Internet, the primary task is to identify data about the city in which the video was filmed, and subsequently the direct location of the shooting. Determining the identities of the participants in the shooting and searching for them will provide information about the circumstances of what is happening. Among them can be both persons directly involved in the frame, and those who ended up there by chance. So, for example, when filming on the street, these are ordinary passers-by who are witnesses, etc. Being closer or having the opportunity to view the situation from a different angle or perspective, they can provide valuable information regarding the events that occurred, take part in identifying a person or report information about him, in the case when the person he is looking for is familiar to him.

Determining the source and time of video posting on the Internet

The importance of finding a video source for personal identification plays a vital, and sometimes decisive, role. In this case, it is necessary to find out both the type of the data source on the Internet and the time of its placement. To do this, you need to establish the domain name of the site to which the first publication belongs, excluding all information that was subsequently replicated as a result of numerous reposts. To compare and search for similar videos, the current services of all available search engines are available. As a rule, sites allow you to identify the name of the author and the time of posting, which can be a fake name or nickname, which, of course, is of no interest without an additional search. When searching for data via the Internet, specialists have the opportunity to use services to search for hosting and site registration; information about possible contacts or its affiliation may also be available on the resource itself.

Search for video author

To find the author of the video, who may have important information, various techniques are used, from IT technologies to reviewing publications and registrations through communication with resource owners and moderators. Every Internet user has access to general information about the ownership of resources, called Whois. By establishing the time and date of content posting, questions and suspicions regarding possible human involvement in the recording can be addressed. Information about video placement is stored in search engines, from where specialists can obtain data, even about those files that have been deleted.

Special algorithms for working with video

Currently, technical means allow for a lot of manipulation when working with video content and files. For the old data format, digitization is provided, which involves analog-to-digital conversion of data for working with them using software. For primitive resolution changes, the video stream can be subject to extrapolation. However, without the use of additional smoothing and filtering technologies, this kind of technique is an ineffective means when it is necessary to obtain detailed visual information and identify a person. To obtain detailed information, change image scales, and take screenshots, specialists use a storyboard. Some algorithms for clearing image noise include storyboarding and working on each frame to reduce the intensity of noise and increase the clarity and quality of the video. Various manipulations with video clips now make it possible to widely use software tools both for various kinds of animation, through which convincing fakes can be created, and for exposing them. Specialists from the DASC detective agency will be able to distinguish a genuine video from an already processed image, which can be retouched in the required manner. Dynamic video and short excerpts are processed in detail by applying tracking masks using various relevant mathematical models. The video sequence is divided into fragments, for which noise and lighting histograms and color balance diagrams are constructed, which can be adjusted if necessary. At the same time, a mathematical model in the form of a tracking algorithm, which functionally allows you to track the slightest movements and movements of objects. Most software that has increased efficiency for implementing work at this stage is paid. Experience and qualifications are also necessary factors for improving the quality of a video stream through technical means.

Finding the continuation of the video if it is not complete

When decoding a video and identifying individuals on it, it is necessary to take into account such a factor as the usefulness of the captured fragment of interest. For example, videos found on the Internet are often incomplete and may end abruptly. Thus, the search for a continuation of the video is relevant. Their content can overlap and be relevant, while staged stories are distinguished by the presence of the same characters, which reduces the search range and allows you to find out additional information. By finding the continuation of the video information, additional data can be obtained, which will become key in the search for the truth, and will also allow one to reliably identify the person of interest.

How to use the service

Search for videos through a web form on the site

To do this, you need to upload a video fragment or image into the appropriate field of the form.

It should be noted that in the case of downloading a video fragment, the server will first be engaged in storyboarding the downloaded video and processing it, which will require additional time.

The page with the result contains the name of the film, the name of the director, information about the country of origin, year of release, genre, names of actors, a brief description, duration of the video, as well as the position found in the video, links to additional information and a tabular image of frames from the video.

Search videos using the app

Searching for videos using the application is much faster, since all pre-processing is done on the client side, and only a small part of the original data is sent to the server. This loads the channel less and increases the speed of search query execution.

Is it possible to populate the contents of a database with index information about one million videos single-handedly?

Most likely no.
Where can I get these videos? How to pump them over the network? Where can I get the computing resources to process them? But you can make the database available for filling by users themselves. And this has already been implemented. We have filled the database with one hundred videos and you can verify the functionality of our service. You can also download and install a free application for indexing videos and adding descriptions, followed by uploading the data to the server. The application allows you to further perform some operations with the downloaded data: deleting, editing the description, viewing and searching.

If you decide to add your video or any other, please first make sure that this video is not yet in the database. You can search in the application by title, director, year of creation and other parameters.

The speed of creating index information depends on the power of your computer and the characteristics of the video itself (resolution, codec, frame rate). On average, processing takes a few minutes. At this time, the user can fill in the video description text fields.

How to find a video from a photo

As you know, today's search engines use special metadata when searching for videos. The latter are text markers that are not displayed on the screen and are intended exclusively for search bots.

External metadata contains information about the author of the video, date of creation, and duration of the video. Internal metadata contains the video title, its description, encoding features and other technical information.

Search engines use the metadata we provide to find relevant images and videos. Typically, when forming search queries, keywords are used that correspond to the content of the video, its authors, the names of the locations where the video was filmed, and so on.

If you only have a photo, from which you cannot understand its subject, location and authors, then it will be necessary to use the functionality of search engines that search for images based on the uploaded picture. Below we will look at how this can be done.

Links

Website

https://www.videocolor.aapsoftware.ru/ The site allows you to search by short video fragment, as well as by image from the video.

Applications

• Windows x64 video identification application Video Color Capture
• Video Color Creator database
• All applications are free.

Video

• Video search by image and video search by short video fragment
• Searching for a movie by video fragment
• Video identification using Video Color Capture
• Video of working with the “Video Color Capture” program version 1.1
• Working with the Video Color Creator program version 2.0

Initial setup of AntiDupl NET

The first thing you need to do after downloading (from the link at the end of the article) the AntiDupl NET program is to configure it...

I recommend deleting duplicates to the trash first - then thank me for this advice.

Be sure to specify a folder to search for similar photos - otherwise the program will spend several hours analyzing absolutely all the images on the computer...