Pathstream

pathstream

As your infrastructure grows, it becomes crucial to have robots and a reliable centralized logging system. Log centralization is becoming a key aspect of a variety of IT tasks and provides you with an overview of your entire system. The best solution is to aggregate the logs from all containers, which is enriched with metadata so that it provides you with better traceability options and comes with awesome community support. This is where ELK Stack comes into the picture. ELK, also known as Elastic stack, is a combination of modern open-source tools like ElasticSearch, Logstash, and Kibana. It is a complete end-to-end log analysis solution you can use for your system. Each component has its defined role to play: ElasticSearch is best in storing the raw logs, Logstash helps to collect and transform the logs into a consistent format, and Kibana adds a great visualization layer and helps you to manage your system in a user-friendly manner. In this guide, you will learn how to deploy ELK and start aggregating container logs. Here we are going to combine ELK with to aggregate the container logs. For this, we are going to build a custom Docker image. Filebeat Step 1 - Configuring Filebeat: Let’s begin with the Filebeat configuration. First, you have to create a Dockerfile to create an image: $ mkdir filebeat_docker && $ touch Dockerfile && nano Dockerfile cd $_ Now, open the in your preferred text editor, and copy/paste below mentioned lines: Dockerfile FROM docker.elastic.co/beats/filebeat:7.5.1
 
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN mkdir /usr/share/filebeat/dockerlogs
RUN chown -R root /usr/share/filebeat/
RUN chmod -R go-w /usr/share/filebeat/ In directory, create a filebeat.yml file that contains configuration for Filebeat. For this guide, we are going to use a minimal file. filebeat_docker filebeat.yml filebeat.inputs:
  - : docker
    containers:
      path: stream: ids:
        - cri.parse_flags: combine_partial: exclude_files: [ ]
 
processors:
  - add_docker_metadata:
      host: filebeat.config.modules:
  path: /modules.d/*.yml
  reload.enabled: output.logstash:
  hosts: [ ] files:
logging.level: error
logging.to_files: logging.to_syslog: loggins.metrice.enabled: logging.files:
  path: /var/ /filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
ssl.verification_mode: none type "/usr/share/dockerlogs/data" "stdout" "*" true true '\.gz$' "unix:///var/run/docker.sock" ${path.config} false "127.0.0.1:5044" log false false false log Now, it’s time to create the Filebeat Docker image: $ docker build -t filebeatimage .
Sending build context to Docker daemon  3.584kB
Step 1/6 : FROM docker.elastic.co/beats/filebeat:7.5.1
7.5.1: Pulling from beats/filebeat
c808caf183b6: Already exists 
a07383b84bc8: Pull complete 
a3c8dd4531b4: Pull complete 
5547f4a87d0c: Pull complete 
d68e041d92cd: Pull complete 
7cfb3f76a272: Pull complete 
748d7fe7bf07: Pull complete 
Digest: sha256:68d87ae7e7bb99832187f8ed5931cd253d7a6fd816a4bf6a077519c8553074e4
Status: Downloaded newer image docker.elastic.co/beats/filebeat:7.5.1
 ---> 00c5b17745d1
Step 2/6 : COPY filebeat.yml /usr/share/filebeat/filebeat.yml
 ---> f6b75829d8d6
Step 3/6 : USER root
 ---> Running 262c41d7ce58
Removing intermediate container 262c41d7ce58
 ---> 1ffcda8f39cf
Step 4/6 : RUN mkdir /usr/share/filebeat/dockerlogs
 ---> Running 8612b1895ac7
Removing intermediate container 8612b1895ac7
 ---> 483d29e65dc7
Step 5/6 : RUN chown -R root /usr/share/filebeat/
 ---> Running 4a6ad8b22705
Removing intermediate container 4a6ad8b22705
 ---> b779a9da7ac9
Step 6/6 : RUN chmod -R go-w /usr/share/filebeat/
 ---> Running bb9638d12090
Removing intermediate container bb9638d12090
 ---> 85ec125594ee
Successfully built 85ec125594ee
Successfully tagged filebeatimage:latest for in in in in To verify if the image was built successfully: $ docker images
REPOSITORY      TAG           IMAGE ID            CREATED             SIZE
filebeatimage   latest        85ec125594ee        7 seconds ago       514MB For container, you have created two mounts using the parameter -v; filebeat_elk : You have mapped host machine docker logs which resides in to inside the docker container. Note that you have used :ro which denotes that has read-only permission. /var/lib/docker/containers:/usr/share/dockerlogs/data /var/lib/docker/containers /usr/share/dockerlogs/data Whereas, is bind with Filebeat container’s Docker daemon, which allows Filebeat container to gather the Docker’s metadata and container logs entries. /var/run/docker.sock Filebeat installation via DEB: There is an alternate way to install Filebeat in your host machine. At the time of writing, Filebeat version is you can download the latest version of filebeat from . 7.5.1 here To install the downloaded file: .deb $ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-amd64.deb

$ sudo dpkg -i filebeat-7.5.1-amd64.deb You can find the configuration file in /etc/filebeat/filebeat.yml directory. Step 2 - Configuring ELK or Elastic Stack You can either use a remote server to host your ELK stack or can launch containers within your existing system. Before you get going, make sure that the following ports are listening: Elasticsearch - Port 9200 and Port 9300 Logstash - Port 5044 Kibana - Port 5601 ElasticSearch: We are going to use the latest official image of Elasticsearch as of now. So begin by pulling the image from Docker Hub: $ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.1
7.5.1: Pulling from elasticsearch/elasticsearch
c808caf183b6: Already exists 
05ff3f896999: Pull complete 
82fb7fb0a94e: Pull complete 
c4d0024708f4: Pull complete 
136650a16cfe: Pull complete 
968db096c092: Pull complete 
42547e91692f: Pull complete 
Digest: sha256:b0960105e830085acbb1f9c8001f58626506ce118f33816ea5d38c772bfc7e6c
Status: Downloaded newer image docker.elastic.co/elasticsearch/elasticsearch:7.5.1
docker.elastic.co/elasticsearch/elasticsearch:7.5.1 for Now, create a directory name as docker_elk, where all your configuration files and Dockerfile will reside: $ mkdir docker_elk && cd $_ Inside , create another directory for and create a and files: docker_elk elasticsearch Dockerfile elasticsearch.yml $ mkdir elasticsearch && $ touch Dockerfile && touch elasticsearch.yml cd $_ Open file in your preferred text editor and copy the configuration setting as it is: elasticsearch.yml ---
cluster.name: network.host: 0.0.0.0
 
xpack.license.self_generated.type: basic
xpack.security.enabled: xpack.monitoring.collection.enabled: "docker-cluster" true true Note that you can set from to if you wish to evaluate the commercial feature of x-pack for 30 days. xpack.license.self_generated.type basic trial Open Dockerfile in your preferred text editor and copy the below-mentioned lines and paste it as it is: FROM docker.elastic.co/elasticsearch/elasticsearch:7.5.1
COPY --chown=elasticsearch:elasticsearch ./elasticsearch.yml /usr/share/elasticsearch/config/ The command is to change the file owner to as of other files in container. chown elasticsearch Kibana: Now, you are going to setup Dockerfile for Kibana, and again you have to pull the latest image from the Elastic Docker registry: $ docker pull docker.elastic.co/kibana/kibana:7.5.1
7.5.1: Pulling from kibana/kibana
c808caf183b6: Already exists 
e12a414b7b04: Pull complete 
20714d0b39d8: Pull complete 
393e0a5bccf2: Pull complete 
b142626e938b: Pull complete 
b28e35a143ca: Pull complete 
728725922476: Pull complete 
96692e1a8406: Pull complete 
e4c3cbe1dbbe: Pull complete 
bb6fc46a19d1: Pull complete 
Digest: sha256:12b5e37e0f960108750e84f6b2f8acce409e01399992636b2a47d88bbc7c2611
Status: Downloaded newer image docker.elastic.co/kibana/kibana:7.5.1
docker.elastic.co/kibana/kibana:7.5.1 for Inside your , create a directory, and inside of it, you have to create a and files: docker_elk Dockerfile kibana.yml $ mkdir kibana && $ touch Dockerfile && touch kibana.yml cd $_ will consist of follow configurations. Note that you have to change the values of and : kibana.yml elasticsearch.user elasticsearch.password ---
server.name: kibana
server.host: elasticsearch.hosts: [ ]
xpack.monitoring.ui.container.elasticsearch.enabled: elasticsearch.username: elastic
elasticsearch.password: yourstrongpasswordhere "0" "http://elasticsearch:9200" true Whereas, in , will look something like this: Dockerfile FROM docker.elastic.co/kibana/kibana:7.5.1
COPY ./kibana.yml /usr/share/kibana/config/ Logstash: Container image for Logstash is available from the Elastic Docker registry. Again at the time of writing current version is 7.5.1, you can find latest version of Logstash . here $ docker pull docker.elastic.co/logstash/logstash:7.5.1
7.5.1: Pulling from logstash/logstash
c808caf183b6: Already exists 
7c07521065ed: Pull complete 
d0d212a3b734: Pull complete 
418bd04a229b: Pull complete 
b22f374f97b1: Pull complete 
b65908943591: Pull complete 
2ee12bfc6e9c: Pull complete 
309701bd1d88: Pull complete 
b3555469618d: Pull complete 
2834c4c48906: Pull complete 
bae432e5da20: Pull complete 
Digest: sha256:5bc89224f65459072931bc782943a931f13b92a1a060261741897e724996ac1a
Status: Downloaded newer image docker.elastic.co/logstash/logstash:7.5.1
docker.elastic.co/logstash/logstash:7.5.1 for Now, create a directory for Logstash inside and add necessary files as shown below: docker_elk $ mkdir logstash && $ touch Dockerfile && touch logstash.yml cd $_ Copy below mentioned line into logstash.yml. Make sure that you enter the right username and password in and respectively: xpack.monitoring.elasticsearch.username xpack.monitoring.elasticsearch.password ---
http.host: xpack.monitoring.elasticsearch.hosts: [ ]
 
xpack.monitoring.enabled: xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: yourstrongpasswordhere "0.0.0.0" "http://elasticsearch:9200" true Now, add following lines into your : Dockerfile FROM docker.elastic.co/logstash/logstash:7.5.1
COPY ./logstash.yml /usr/share/logstash/config/
COPY ./logstash.conf /usr/share/logstash/pipeline/ Apart from this, you have to create a file. Here in elasticsearch reference you will find , and , make sure you change the values as per your system: logstash.conf host user password input {
    tcp {
    port => 5000
    codec => json
  }
}
output {
  elasticsearch {
    hosts => user => elastic
    password => yourstrongpasswordhere
  }
} "elasticsearch:9200" As you are through with the setup of your stack's components, the directory structure of your project should should look something like this: .
├── elasticsearch
│   ├── Dockerfile
│   └── elasticsearch.yml
├── kibana
│   ├── Dockerfile
│   └── kibana.yml
└── logstash
    ├── Dockerfile
    ├── logstash.conf
    └── logstash.yml
 
3 directories, 7 files Now, it’s time to create a Docker Compose file, which will let you run the stack. Step 3 - Docker Compose Create a file in the directory. Here you are going to define and run your multi-container application consist of Elasticsearch, Kibana, and Logstash. docker-compose.yml docker_elk You can copy the below-mentioned context in your docker-compose.yml file. Please make sure that you change the and values. For this guide, is set to 256 MB, but in real world scenarios you might want to increase the heap size as per requirement. ELASTIC_PASSWORD ES_JAVA_OPTS ES_JAVA_OPTS version: services:
 elasticsearch:
   build:
     context: elasticsearch/
   volumes:
     - : volume : elasticsearch
       target: /usr/share/elasticsearch/data
   ports:
     - - environment:
     ES_JAVA_OPTS: ELASTIC_PASSWORD: yourstrongpasswordhere
     discovery.type: single-node
   networks:
     - elk_stack
 
 logstash:
   build:
     context: logstash/
   ports:
     - - environment:
     LS_JAVA_OPTS: networks:
     - elk_stack
   depends_on:
     - elasticsearch
 
 kibana:
   build:
     context: kibana/
   ports:
     - networks:
     - elk_stack
   depends_on:
     - elasticsearch
 
networks:
 elk_stack:
   driver: bridge
 
volumes:
 elasticsearch: '3.2' type source "9200:9200" "9300:9300" "-Xmx256m -Xms256m" "5000:5000" "9600:9600" "-Xmx256m -Xms256m" "5601:5601" Now, to build the ELK stack, you have to run the following command in your directory: docker_elk $ docker-compose up -d
Starting elastic_elk ... Starting kibana_elk   ... Starting logstash_elk ... done done done To ensure that the pipeline is working all fine, run the following command to see the Elasticsearch indices: $ curl -u elastic:yourstrongpasswordhere

health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .triggered_watches                m-l01yMmT7y2PYU4mZ6-RA   1   0          0            0      6.5kb          6.5kb
green  open   .watcher-history-10-2020.01.10    SX3iYGedRKKCC6JLx_W8fA   1   0       1523            0        2mb            2mb
green  open   .management-beats                 ThHV2q9iSfiYo__s2rouIw   1   0          6            1     40.5kb         40.5kb
green  open   .ml-annotations-6                 PwK7Zuw7RjytoWFuCCulJg   1   0          0            0       283b           283b
green  open   .monitoring-kibana-7-2020.01.10   8xVnx0ksTHShds7yDlHQvw   1   0       1006            0    385.4kb        385.4kb
green  open   .monitoring-es-7-2020.01.10       CZd89LiNS7q-RepP5ZWhEQ   1   0      36412          340     16.4mb         16.4mb
green  open   .apm-agent-configuration          e7PRBda_QdGrWtV6KECsMA   1   0          0            0       283b           283b
green  open   .ml-anomalies-shared              MddTZQ7-QBaHNTSmOtUqiQ   1   0          1            0      5.5kb          5.5kb
green  open   .kibana_1                         akgBeG32QcS7AhjBOed3LA   1   0       1105           28    687.1kb        687.1kb
green  open   .ml-config                        CTLI-eNdTkyBmgLj3JVrEA   1   0         22            0     56.6kb         56.6kb
green  open   .ml-state                         gKx28CMGQiuZyx82bNUoYg   1   0          0            0       283b           283b
green  open   .security-7                       krH4NlJeThyQRA-hwhPXEA   1   0         36            0     83.6kb         83.6kb
green  open   .logstash                         7wxswFtbR3eepuWZHEIR9w   1   0          0            0       281b           281b
green  open   .kibana_task_manager_1            ft60q2R8R8-nviAyc0caoQ   1   0          2            1     16.2kb         16.2kb
yellow open   filebeat-7.5.1-2020.01.10-000001  1-RGhyG9Tf-wGcepQ49mmg   1   1          0            0       283b           283b
green  open   .monitoring-alerts-7              TLxewhFyTKycI9IsjX0iVg   1   0          6            0     40.9kb         40.9kb
green  open   .monitoring-logstash-7-2020.01.10 dc_S5BhsRNuukwTxbrxvLw   1   0       4774            0      1.1mb          1.1mb
green  open   .watches                          x7QAcAQZTrab-pQuvonXpg   1   0          6            6    120.2kb        120.2kb
green  open   .ml-notifications-000001          vFYzmHorTVKZplMuW7VSmw   1   0         52            0     81.6kb         81.6kb 'localhost:9200/_cat/indices?v' Now, it is time to pay a visit to our Kibana dashboard. Open your browser and enter the URL . Now enter the predefined username and password; in our case, it is and , respectively. http://your-ip-addr-here:5601 elastic yourstrongpasswordhere In your Kibana dashboard, go to the Management tab, and under Kibana, click on . In the first row, you will find the index, which already has been identified by Kibana. Index Patterns filebeat-* Now, go to the Discover tag on the Kibana dashboard and view your container logs along with the metadata under the selected index pattern, which could look something like this: Conclusion: You have now installed and configured the ELK Stack on your host machine, which is going to collect the raw log from your Docker into the stack that later can be analyzed or can be used to debug applications. About the author - Sudip is a Solution Architect with more than 15 years of working experience, and is the founder of Javelynn . He likes sharing his knowledge by regularly writing for Hackernoon , DZone , Appfleet and many more. And while he is not doing that, he must be fishing or playing chess. Previously posted at https://appfleet.com/ .

Docker Centralized Logging with ELK Stack [EXPLAINED]

There are a lot of exciting and valuable Computer Vision tasks. For instance, Let’s say we have a project where we want to search for lost people in the forest using camera-fitted drones and Computer Vision. Or maybe we need to find something small and we have a high-quality camera. In cases like these, we can get high-resolution images in our dataset. However, most CV models will lower image resolution because it helps with speed (both on train and inference) and usually, you don't need very high resolutions to detect an object. In our case, however, we really need full resolution, because we would be using camera-fitted drones for our search. Meaning that even people would appear as tiny objects in our images. How do we go about a task like this? Dataset Everything starts with a dataset. I've tried to find an open dataset and ended up using . This is not a perfect dataset for searching for lost people, but we will use it as an example. TinyPerson I've converted the dataset from format to , as I was going to use a model from the YOLO family. Furthermore, I've filtered the dataset a little bit, but nothing really important. To convert the dataset, I used and customized it a tiny bit, so it would work in my case. My final dataset consists of 1495 images. I've split it into train/val, where 15% was left for validation. COCO YOLO format this repo A few notes on this Dataset. Images are hard, there can be a lot of people on it, and they can be really far away. It's not rare when not everyone is labeled, which is not good. Anyway, here is an ‘ok’ example from the dataset: It’s important to keep in mind that there are not a lot of high-res images in this dataset, which is unfortunate. The median size is 1920x1080. Baseline As a baseline model, I took pre-trained , which uses 1280x1280 images and is a pretty beefy model. I've also tried with the same image size and similar model size, but I got worse results on this dataset, so I settled on YOLOv5. YOLOv5l6 YOLOv7-W6 Having the correct dataset, it's easy to train Yolov5 with commands like this one: python train.py --data dataset/dataset.yaml --weights yolov5m6.pt --img 1280 --batch 15 --epochs 80 I got mAP50 around for my baseline. You that on this dataset that's an ok mAP. 0.477 can see So we still lose some data in most cases in our baseline, as we resize every image to 1280x1280. When our objects are that small, we don't really want to do that. It would be better to preserve full resolution. But, models are optimized for their training image size, so we need to use 1280x1280 or 640x640 in the case of the YOLO family. Patch-based approach This is another approach that could help us. What if we take every image and cut it with a fixed-size sliding window and use that crop as training data? Let's take a smaller model with an input size of 640x640 and cut our dataset to that size. Here is how I see it: We'll use an image with a size of 1920x1080 as an example. We can divide it into 6 images (patches), 640x640. We can't slice it exactly, we will have intersections and that's fine: Blue boxes are first-row patches, and green ones are second. We don't have an intersection between columns here, just because 1920 is divisible by 640, but we do have an intersection between the first and second rows. With this intersection, we lower the chance that our object will get cut on the edge of the patch, although it is already small because our objects are small. And don't forget that you need to carry out the same process with your labels. Anyway, we can use this approach for any image, no matter the size (if it is bigger than our patch size). This allows us: Not to lose image resolution To use optimal input size for the model, no matter what image sizes we originally have This approach is also good if you don't have enough memory on your GPU. You can find yourself in that situation with 3D really often. is a good lib for that tasks. But in our case, I decided to use a simple custom patcher, specifically written for YOLO models. TorchIO Here is my utils.py: from typing import List

def xywh_to_xyxy(
    lines: List[str],
    img_height: int,
    img_width: int) -> List[List[int]]:

    '''
    This function gets list with YOLO labels in a format:
    label, x-center, y-center, bbox width, bbox height
    coordinates are in relative scale (0-1).
    Returns list of lists with xyxy format and absolute scale.
    '''
    labels = []
    for _, cur_line in enumerate(lines):
        cur_line = cur_line.split(' ')
        cur_line[-1] = cur_line[-1].split('\n')[0]

        # convert from relative to absolute scale (0-1 to real pixel numbers)
        x, y, w, h = list(map(float, cur_line[1:]))
        x = int(x * img_width)
        y = int(y * img_height)
        w = int(w * img_width)
        h = int(h * img_height)

        # convert to xyxy
        left, top, right, bottom = x - w // 2, y - h // 2, x + w // 2, y + h // 2
        labels.append([int(cur_line[0]), left, top, right, bottom])

    return labels


def xyxy_to_xywh(
    label: List[int],
    img_width: int,
    img_height: int) -> List[float]:

    '''
    This function gets list with label and coordinates in a format:
    label, x1, y1, x2, y2
    coordinates are in absolute scale.
    Returns list with xywh format and relative scale
    '''

    x1, y1, x2, y2 = list(map(float, label[1:]))
    w = x2 - x1
    h = y2 - y1

    x_cen = round((x1 + w / 2) / img_width, 6)
    y_cen = round((y1 + h / 2) / img_height, 6)
    w = round(w / img_width, 6)
    h = round(h / img_height, 6)

    return [label[0], x_cen, y_cen, w, h] And here is the patcher itself: class Patcher:
    def __init__(self, path_to_save: Union[Path, str], base_path: Union[Path, str]) -> None:
        self.path_to_save = path_to_save
        self.create_folders()
        self.base_path = base_path

    def create_folders(self) -> None:
        self.path_to_save.mkdir(parents=True, exist_ok=True)
        (self.path_to_save / "images").mkdir(exist_ok=True)
        (self.path_to_save / "labels").mkdir(exist_ok=True)

    def patch_sampler(
        self,
        img: np.ndarray,
        fname: str,
        patch_width: int = 640,
        patch_height: int = 640,
    ) -> None:
        # Get image size and stop if it's smaller than patch size
        img_height, img_width, _ = img.shape
        if img_height < patch_height or img_width < patch_width:
            # copy image and label to patched folder
            io.imsave(self.path_to_save / "images" / f"{fname}.jpg", img)
            label_path = (self.base_path / "labels" / fname).with_suffix(".txt")
            copy(label_path, self.path_to_save / "labels")
            return

        # Get number of horisontal and vertical patches
        horis_ptch_n = int(np.ceil(img_width / patch_width))
        vertic_ptch_n = int(np.ceil(img_height / patch_height))
        y_start = 0

        ##### Prepare labels
        label_path = (self.base_path / "labels" / fname).with_suffix(".txt")
        with open(label_path) as f:
            lines = f.readlines()

        all_labels = xywh_to_xyxy(lines, *img.shape[:2])
        #####

        # Run and create every crop
        for v in range(vertic_ptch_n):
            x_start = 0

            for h in range(horis_ptch_n):
                idx = v * horis_ptch_n + h

                x_end = x_start + patch_width
                y_end = y_start + patch_height

                # Get the crop
                cropped = img[y_start:y_end, x_start:x_end]

                ##### Get labels patched
                cur_labels = []
                for label in all_labels:
                    cur_label = label.copy()

                    # Check if label is insde the crop
                    if (
                        label[1] > x_start
                        and label[2] > y_start
                        and label[3] < x_end
                        and label[4] < y_end
                    ):
                        # Change scale from original to crop
                        cur_label[1] -= x_start
                        cur_label[2] -= y_start
                        cur_label[3] -= x_start
                        cur_label[4] -= y_start

                        label_yolo = xyxy_to_xywh(cur_label, patch_width, patch_height)
                        cur_labels.append(label_yolo)

                # Save the label file to the disk
                if len(cur_labels):
                    with open(self.path_to_save / "labels" / f"{fname}_{idx}.txt", "a") as f:
                        f.write("\n".join("{} {} {} {} {}".format(*tup) for tup in cur_labels))
                        f.write("\n")
                #####

                # Save the crop to disk
                io.imsave(self.path_to_save / "images" / f"{fname}_{idx}.jpg", cropped)

                # Get horisontal shift for the next crop
                if img_width % patch_width == 0:
                    x_start += patch_width
                else:
                    x_start += int(
                        patch_width
                        - (patch_width - img_width % patch_width) / (img_width // patch_width)
                    )

            # Get vertical shift for the next crop
            y_start += int(
                patch_height
                - (patch_height - img_height % patch_height) / (img_height // patch_height)
            )


def main():
    '''
    base path structure:

    -> dataset
    ---> train
    -----> images (folder with images)
    -----> labels (folder with labels)
    ---> valid
    -----> images (folder with images)
    -----> labels (folder with labels)
    '''

    base_path = Path("")

    # path were you want to save patched dataset
    path_to_save = Path("")

    for split in ["train", "valid"]:
        images_folder_path = base_path / split / "images"

        patcher = Patcher(path_to_save / split, base_path / split)

        for image_path in images_folder_path.glob("*"):
            if image_path.name.startswith("."):
                continue
            image = io.imread(image_path)
            fname = image_path.stem

            patcher.patch_sampler(image, fname)


if __name__ == "__main__":
    main() Training YOLOv5 is still easy: python train.py --data dataset/dataset.yaml --weights yolov5m.pt --img 640 --batch 40 --epochs 80 mAP increased a little bit - to : 0.499 So we have increased our mAP by , which is not much, and our new solution is going to be a little bit slower (both on inference and while training). But this technique is going to perform better, the higher the resolution of the images you have. And we can see a difference even with our image size. 4% Conclusion The best use case for this technique is when you don't mind losing some speed to gain accuracy, and you have high-res images with small objects. In other similar cases, you can also try this solution after you have a baseline. Don't forget to split the dataset first and sample patches second, so you don't create a data leakage.

Small Object Detection in Computer Vision: The Patch-Based Approach

For several weeks, French Internet users have reported difficulty accessing dozens of unlicensed streaming and torrent sites. It now transpires that following legal action by several movie industry groups, a Paris court handed down a ruling ordering some of the country's top ISPs to prevent their customers from accessing around 36 platforms. In common with at least 30 countries all around the world, France is becoming ever more receptive of site-blocking as a means to mitigate copyright infringement. France has been blocking ‘pirate’ sites for more than five years on behalf of the music and movie industries but last year academic publishers Elsevier and Springer Nature also got involved. As a result, a court in Paris several of the largest French ISPs to block access to 57 domain names related to pirate scientific libraries LibGen and Sci-Hub. ordered Later that year, France broke fresh ground again. Following a complaint from anti-piracy group SCPP, a court handed down the compelling local ISPs to block file-hosting services. very first European order Several weeks ago, Internet users in France reported fresh difficulties accessing many ‘pirate’ sites, mainly platforms that in some way offer access to streaming movies and TV shows. It now transpires that a new wave of blocking has hit the country following legal action initiated by movie industry groups. According to NextInpact , the National Federation of Film Publishers (FNEF), the Syndicate of Digital Video Editing (SEVN), the Union of Film Producers (UPC) and the National Cinema Center (CNC) teamed up back in October 2019 to request the blocking of dozens of pirate sites. ( paywall ) Reports on the sites’ activities were compiled by local anti-piracy group ALPA, which noted that torrent site YTS reportedly made available 11,676 movies without permission from rightsholders, “the vast majority of which are legally available in theaters, digital or online.” TV torrent index EZTV was accused of offering 4,781 on the same basis. Streaming platform Allostream reportedly offered more than 1,300 movies and close to 900 TV shows. Overall, it was estimated that almost three-quarters of the content on the site was pirated. When compared to the other sites this was relatively low, however, with most others clocking up 85% unlicensed copyrighted content. This evidence was enough to convince the court. “[The rightsholders] have established with sufficient evidence that the disputed sites allow internet users, via the aforementioned paths, to download or continuously access works protected works via hypertext links, without having the authorization of the rights holders, which constitutes an infringement of copyright or neighboring rights,” the court ruling reads, per NextInpact. In common with similar blocking applications around Europe, the French court considered not only the intellectual property rights of the movie companies but also the rights of the Internet service providers to do business and their customers’ rights to send and receive information on the Internet. In the end, the rights of the content companies outweighed the interests of the other parties. As a result, the Paris court handed down an order on January 16, 2020, which compelled ISPs including Bouygues Télécom, Free, Orange, and SFR to prevent their users from accessing 36 platforms listed in the complaint. The vast majority appear to be streaming-related sites with the addition of torrent indexes YTS and EZTV. (full list below) While there are 36 headline sites, the court recognized that these platforms can operate from many more domains. In respect of TV torrent platform EZTV, for example, the site’s .io, .it, .ch, .re and .ag domains are listed for blocking. For movie index YTS, three domains appear – yts.lt, .am and .ag. As a result, 79 domains are detailed overall in the order handed down by the Paris court, which is valid for 18 months. Despite the comprehensive attempt at covering all domains, it’s likely the platforms will seek out new ones in response to the order. Should evasive action be taken by the sites, the complainants will have to obtain permission from a judge to update the list of domains. The full list of sites blocked by ISPs following the latest order is as follows: Allostream Annuaire-telechargement-ec Annuaire-telechargement-fr ATFUT DivxTOP DPSTREAM DPStreaming ETOPOP EZTV Filme-Streaming Filmcomplet FilmStreamingg1 FilmStreaming1FV Filmz FRStream Full-Serie HDS-Streaming HDSS LibertyVF N1Streaming Papystreaming Planet-Streaming1 Radego Serie-Streaming Seriecomplete SKStream Streamcomplet Streamdirect Streaming-VOSTFR StreamingDIVX1 Time2watch VKStreaming Voir-Films-Series VOSTFRSerie Wikiserie YTS The related court order can be found here (pdf) Originally published as “ French ISPs Block Dozens of Pirate Sites Following Movie Industry Action ” with the Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license

Stories, News & Mentions about PATHSTREAM

Pathstream’s Stories on HackerNoon