Streamkapis in the Business Intelligence Industry

Streamkap

Who is this article for? The material is designed for developers with little experience with , and the main reason is to refresh their knowledge. Java Table of Contents: Variables and types It will be better to use these Switch and case. Try, catch, and finally. Lambdas, streams, optional? Record class Variables and types. Let’s start with the general and important things that you really should have learned earlier. Java, like many languages, has two types of variables: Primitives: Non-Primitive: Classes and arrays! Usually, we need to compare variables, but what methods should we use? “equals” or “== “ ? Let’s look at all the comparison methods available to us: ==      equal to
!=      not equal to
>       greater than
>=      greater than or equal to
<       less than
<=      less than or equal to This tricky question was asked in the interview. We can use it with classes, but: // Integer != int
int a = 1, b = 1;
System.out.println(a == b); // true

int c = 1, j = c;
System.out.println(c == j); // true

Integer ai = Integer.valueOf(1), bi = Integer.valueOf(1);
System.out.println(ai == bi); // true 

Integer ci = Integer.valueOf(128), ji = Integer.valueOf(128);
System.out.println(ci == ji); // false
/* You can always count on the fact that 
*  for values between -128 and 127, 
*  you get the identical Integer objects after autoboxing, 
*  and on some implementations you might get identical objects
*  even for higher values.
*/ It’s important to learn that the Integer class is not a primitive value because it is a regular class. Class is the root of the class hierarchy. Every class has as a superclass. All objects, including arrays, implement the methods of this class. Object Object The “==” operator compares the values of two object references to see if they refer to the same thing. For an object, we should use “equals”: // Indicates whether some other object is "equal to" this one.
public boolean equals(Object obj) Or interface for natural ordering (if you need to sort your collection with your own type of object): Comparable public class HugePoint implements Comparable<HugePoint> {

    // same as before

    @Override
    public int compareTo(HugePoint otherHugePoint) {
        return Integer.compare(getSize(), otherHugePoint.getSize());
    }

} The interface defines a method: Comparator compare(arg1, arg2) public class HugePointComparator implements Comparator<HugePoint> {

    @Override
    public int compare(HugePoint firstPoint, HugePoint secondPoint) {
       return Integer.compare(firstPoint.getSize(), secondPoint.getSize());
    }

} It’s easy to use Java 8+ this way: Comparator<Person> byAge = Comparator.comparing(Person::getAge);

Comparator byAge = (Person person1, Person person2) -> 
    Integer.compare(person1.getAge(), person2.getAge()); Which one do you need to use and when? I would recommend using “Comparable” interface for ordering (if you have a sorting method, etc.), and “Comparator” if you need to compare objects by a special field. It will be better to use these Date and time Why? For the current timestamp, just use . We don’t need to convert to milliseconds. Instant.now() // Instant class, one of the main classes of the Date-Time API, encapsulates a point on the timeline.
Instant.now();
Instant now = Instant.parse("2021-02-09T11:19:42.12Z");
Instant before = now.minus(Duration.ofDays(1)); For the local date format use the following: // LocalDateTime is an immutable date-time object 
// that represents a date-time, often viewed 
// as year-month-day-hour-minute-second. 
LocalDateTime.now();
LocalDateTime.of(2001, Month.FEBRUARY, 10, 03, 01); UUID // uuid
UUID uuid = UUID.randomUUID();
UUID uuid = new UUID(mostSignificant64Bits, leastSignificant64Bits); Random // random for int
int randomWithMathRandom = (int) ((Math.random() * (max - min)) + min);

// or easy way
Random random = new Random();
int randomWithNextInt = random.nextInt(); Switch and case. Try, catch, and finally When we have too many conditions, and they are all tied to the same type of variable in “if-then-else,” it is worth using a switch-case. // case 
int expression = 9;
switch(expression) {
  case 2:
    System.out.println("Small Size");
    break;

  case 3:
    System.out.println("Large Size");
    break;
        
  // default case
  default:
    System.out.println("Unknown Size");
} Also, up Java to version 19 or more, and we can use the expression: Day day = Day.WEDNESDAY;
var resultDay = switch (day) {
        case MONDAY, FRIDAY, SUNDAY -> 6;
        case TUESDAY                -> 7;
        case THURSDAY, SATURDAY     -> 8;
        case WEDNESDAY              -> 9;
        default -> throw new IllegalStateException("Invalid day: " + day);
    } 

System.out.println(resultDay); Don’t forget to use try/catch with methods that can throw exceptions. Try/catch and try with resources: // old
public int find(String playerFile) {
    try {
        Scanner contents = new Scanner(new File(playerFile));
        return Integer.parseInt(contents.nextLine());
    } catch (FileNotFoundException noFile) {
        throw new IllegalArgumentException("Couldn't find the file");
    }
}

// with resources
public int find(String playerFile) {
    try (Scanner contents = new Scanner(new File(playerFile))) {
      return Integer.parseInt(contents.nextLine());
    } catch (FileNotFoundException e ) {
      logger.warn("Couldn't find the file");
      return 0;
    }
}

// don't do it!
public int find(String playerFile) {
    try {
        // action
    } catch (Exception e) {} // <== catch and swallow
    return 0;
} Use , especially with channels: finally // close it!
public int find(String playerFile) throws FileNotFoundException {
    Scanner contents = null;

    try {
        contents = new Scanner(new File(playerFile));
        return Integer.parseInt(contents.nextLine());
    } finally {
        if (contents != null) {
            contents.close();
        }
    }
}

// Always close it!
public int find(String playerFile){
    MessageChannel channel = buildChannel(port);
    try {
      // do some action with channel
    } finally {
      channel.close(); // close!
    }
} Lambdas, streams, optional? The main reason for switching to Java 8 and higher is to be able to work with threads, streams, and functional interfaces. In this article, we will cover only working with streams, and we will deal with threads on the next page. First point — collections: Also, it would be good to go through the algorithms’ complexity table. I strongly recommend using this link: or the full article . gist here Let’s analyze the two approaches and see the difference. // old way: 
List<String> old = new ArrayList<>();
old.add("one");

// new way:
List<String> list = Arrays.asList("one", "two");
List<String> list = List.of("one", "two", "three");
Set<String> set = Set.of("one", "two", "three"); We can simplify working with the map. // old:
HashMap<Integer, String> hm = new HashMap<Integer, String>();
hm.put(1, "foo");

// new:
Map<String, String> map = Map.of("foo", "one", "bar", "two"); As you know, we have only two types of operations: and operations. intermediate terminal // Intermediate Operations
var stream = ...;
stream.filter();
stream.map();
stream.flatMap();
stream.distinct();
stream.sorted();
stream.peek();
stream.limit();
stream.skip();


// Terminal
stream.reduce();
stream.forEach();
stream.min();
stream.max();
stream.anyMatch();
stream.allMatch();
stream.noneMatch();
stream.findAny();
stream.findFirst();
stream.toArray();
stream.collect();
stream.count();
stream.forEachOrdered(); Just code: // Java 7 way
ArrayList results = new ArrayList(); 
for (Score score: scores) { 
  if (score.getVal() > 10) { 
      results.add(score.getVal()); 
   } 
} 
System.out.println(results);

// Java 8+ way
ArrayList results = scores.stream()
      .filter(score -> score.getVal() > 10)
      .collect(Collectors.toList()); 

System.out.println(results); If you have a duplicate code, just use functional interfaces: // Functional 
Consumer<Integer> consumer = (n) -> { System.out.println(n); };
numbers.forEach(consumer); Throwing exceptions every time an object is not found is not a good idea; you should use the “Optional” class for this: var empty = Optional.empty();
empty.isPresent(); // false

var notEmpty = Optional.of(name);
notEmpty.isPresent(); // true

String name = null;
Optional<String> opt = Optional.ofNullable(name);
opt.isPresent(); // false

String name = null;
String result = Optional.ofNullable(nullName).orElse("Default");

// get result
var name = Optional.of("Name");
name.get(); And if you really need to throw an exception, you should use “orElseThrow”: String name = Optional.ofNullable(name)
    .orElseThrow(IllegalArgumentException::new);

// throw NoSuchElementException.class
Optional.ofNullable(name).orElseThrow(); Record classes A typical situation: API had been written with “none thread safe” methods, and we didn’t know about the magic inside the library. What exactly do I mean? Any variables are mutable by default, which means we can change the value they hold. We don’t have read-only methods for classes inside; we can change them inside any threads: public class TestObject {
  private String string;
  // public contructor 
  // getters and setters
} In action: // create new object
ExecutorService executor = Executors.newFixedThreadPool(10);
var testObject = new UsefulObject();

executor.submit(() -> testObject.setTestString("test_1"));
executor.submit(() -> testObject.setTestString("test_2"));
executor.submit(() -> testObject.setTestString("test_3"));
executor.submit(() -> testObject.setTestString("test_4")); And what about the result? // the result might be anything!
// test_1 or test_2
// or test_3
System.out.println(testObject.getTestString()); . They are immutable classes and are implicitly classes, which means they can’t be extended. Just use a record class instead of a class: Record classes are special classes that act as transparent carriers for immutable data final record MainObject(int x) { } The compiler implicitly provides the constructor, getters, & , and . equals hashCode toString public final class MainObject {

  private final int x;

  public Rectangle(int x) {
        this.x = x;
  }

  double x() { return this.x; }
  // Implementation of equals() and hashCode(), which specify
  // that two record objects are equal if they
  // are of the same type and contain equal field values.
  public boolean equals...
  public int hashCode...
  // An implementation of toString() that returns a string
  // representation of all the record class's fields,
  // including their names.
  public String toString() {...}
} If we need to share the class now, we can do it without synchronization. That’s it for the first part! Thank you, and good luck! Notes Java primitives https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html Class object https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html How we can work with collections after Java 7: https://docs.oracle.com/javase/8/docs/technotes/guides/collections/overview.html Java intermediate and terminal operation: https://javaconceptoftheday.com/java-8-stream-intermediate-and-terminal-operations/

Java Essentials: A Quick Guide for Refreshing Developers' Knowledge

Kafka is widely used for building event-driven architectures, allowing services to communicate through messages. However, when working with a foreign provider that adds a " " header to messages, you may encounter serialization and deserialization issues. microservices TypeId This article will guide you through the process of configuring a custom type mapper in Spring Kafka to handle the mapping of types between the producer and consumer services. By implementing this approach, you can successfully consume messages with different package structures and types, avoiding exceptions caused by incompatible types and ensuring smooth message processing. Understanding the deserialization Exception with " " Header TypeId In scenarios where the classes used in the consumer service have a different package structure compared to the producer service's DTO classes, the serialization exception can occur due to the mismatched class paths. DTO The serialization process relies on both the producer and the consumer having access to the same class definitions. When the consumer receives a message with a " " header referencing a class that resides in a different package than the consumer's DTO classes, the deserialization process fails to locate the appropriate class definition. This results in the "ClassNotFoundException" or "IllegalArgumentException" within the serialization exception. TypeId These exceptions can disrupt the smooth consumption of messages. Understanding the root cause of this exception is key to effectively handling the situation. An example of the serialization exception is as follows: javaCopy codejava.lang.IllegalStateException: This error handler cannot process 'SerializationException's directly; please consider configuring an 'ErrorHandlingDeserializer' in the value and/or key deserializer
	at org.springframework.kafka.listener.DefaultErrorHandler.handleOtherException(DefaultErrorHandler.java:198) ~[spring-kafka-3.0.7.jar:3.0.7]
Caused by: org.apache.kafka.common.errors.RecordDeserializationException: Error deserializing key/value for partition consumer-test-18 at offset 1. If needed, please seek past the record to continue consumption.
	at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:1438) ~[kafka-clients-3.4.0.jar:na]
Caused by: java.lang.IllegalArgumentException: The class 'org.example.dto.out.OutMessageImplTwo' is not in the trusted packages: [java.util, java.lang, org.example.dto.in, org.example.dto.in.*]. 
If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all (*).
	at org.springframework.kafka.support.mapping.DefaultJackson2JavaTypeMapper.getClassIdType(DefaultJackson2JavaTypeMapper.java:129) ~[spring-kafka-3.0.7.jar:3.0.7] The exception message provides insights into the underlying problem. In this case, the exception occurs because the class is not present in the list of trusted packages. The trusted packages serve as a safeguard to ensure secure deserialization and prevent unauthorized classes from being deserialized. org.example.dto.out.OutMessageImplTwo When deserializing a message, the default behavior of Spring Kafka is to check if the class specified in the "TypeId" header is within the trusted packages. If it is not found, an exception is thrown. This mechanism is in place to protect against potential security risks, as deserializing untrusted classes can lead to code execution vulnerabilities. To resolve this issue, you need to ensure that the consumer can properly map the incoming message to the corresponding class, even if they have different package structures. One approach is to configure a custom type mapper that explicitly handles the type mapping between the " " header and the consumer's DTO classes. TypeId By implementing a custom type mapper, you can instruct the consumer to map the " " header to the appropriate class or even ignore this header. This enables successful deserialization and ensures that the consumer can process the message correctly. TypeId In the upcoming sections, we will delve into the details of configuring and implementing a custom type mapper in Kafka listeners, allowing seamless message processing and addressing the challenges encountered when the consumer's DTO classes differ from those of the producer. Implementation of custom-type mapper In our example, we will send an implementation of OutMessageAbstract to Kafka and will read it as an implementation of the InMessageAbstract class. These classes are absolutely identical, they differ in name and package. The figure below shows the project structure. Our KafkaListener looks like this. It implements the logic for processing each type of message from the topic. @Component
@Log
@KafkaListener(
        topics = "${spring.cloud.stream.bindings.topic-in-0.destination}",
        groupId = "${spring.cloud.stream.bindings.topic-in-0.group}",
        containerFactory = "kafkaListenerContainerFactory")
public class EventListeners {
    @KafkaHandler
    public void handleInMessageImplEvent(@Payload InMessageImpl event) {
        System.out.println(event);
    }

    @KafkaHandler
    public void handleInMessageImplTwoEvent(@Payload InMessageImplTwo event) {
        System.out.println(event);
    }
} We use the kafkaListenerContainerFactory bean to configure our listener. This factory is described in KafkaConsumerConfiguration and looks like this: @Configuration
public class KafkaConsumerConfiguration {
    private final String SERVER;
    private final String SERVER_PORT;

    public KafkaConsumerConfiguration(
            @Value("${spring.cloud.stream.kafka.binder.brokers}") String server,
            @Value("${spring.cloud.stream.kafka.binder.defaultBrokerPort}") String port
    ) {
        this.SERVER = server;
        this.SERVER_PORT = port;
    }

    private <T> ConsumerFactory<String, T> typeConsumerFactory(Class<T> clazz) {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, String.format("%s:%s", SERVER, SERVER_PORT));

        return new DefaultKafkaConsumerFactory<>(
                props,
                new StringDeserializer(),
                new JsonDeserializer<>(clazz));
    }

    private <T> ConcurrentKafkaListenerContainerFactory<String, T> initFactory(Class<T> clazz) {
        ConcurrentKafkaListenerContainerFactory<String, T> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(typeConsumerFactory(clazz));
        return factory;
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, ?> kafkaListenerContainerFactory() {
        return initFactory(InMessageAbstract.class);
    }
} Because we're adding a “Typeld” header to the message, we'll get the error described above when we try to deserialize. However, we can configure the JsonDeserializer manually. To do this, we must define our implementation of TypeMapper. The following shows how you can define your own TypeMappers. class CustomTypeMapper<T> extends DefaultJackson2JavaTypeMapper {
    private final Class<T> clazz;

    public CustomTypeMapper(Class<T> clazz) {
        this.clazz = clazz;
    }

    @Override
    public JavaType toJavaType(Headers headers) {
        try {
            return TypeFactory.defaultInstance()
                    .constructType(ClassUtils.forName(clazz.getName(), getClassLoader()));
        } catch (ClassNotFoundException e) {
            throw new RuntimeException(e);
        }
    }
} If we add our TypeMapper implementation to the JsonDeserializer when we create the factory, then we can force the Json to be deserialized to the class we need, ignoring the header. The corrected method would look like this: private <T> ConsumerFactory<String, T> typeConsumerFactory(Class<T> clazz) {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, String.format("%s:%s", SERVER, SERVER_PORT));

    JsonDeserializer<T> jsonDeserializer = new JsonDeserializer<>(clazz);
    jsonDeserializer.setTypeMapper(new CustomTypeMapper<>(clazz));

    return new DefaultKafkaConsumerFactory<>(
            props,
            new StringDeserializer(),
            jsonDeserializer);
} Conclusion By utilizing a custom type mapper in Spring Kafka, you can overcome the challenges associated with consuming messages from foreign providers that include a " " header. This article has provided a step-by-step guide to configuring and implementing the custom type mapper, enabling smooth message processing and avoiding serialization and deserialization exceptions. With this knowledge, you can confidently integrate with external systems and consume Kafka messages with different package structures. TypeId The full project code is available on GitHub: https://github.com/stepanovD/kafka-templates/tree/master/consumer-custom-type-mapper

Handling Custom Type Mapping in Kafka Listeners for Messages with "TypeId" Header

As Staff Engineer on Lob’s team, I was recently tasked with standing up an endpoint for customers that supplied them with all the data we had stored for a particular zip code. We use pretty heavily to store our data so this boiled down to querying ES by zip code and writing the data to Amazon S3. But both services have subtle limitations on how you read/write them: ES caps your result set to 10,000 records, and Amazon S3 insists that you write to your buckets in chunks of at least 4 Megabytes. This turned out to be a perfect use case for Elixir streams! Address Verification Elasticsearch Elixir Streams Before we dive into the nitty gritty of Elasticsearch and AWS S3, let’s first do a brief refresher on Elixir Streams. What is a Stream? . This is typically accomplished by maintaining a little bit of state that describes where you currently are in your enumerable and a method of computing the next item. A trivial example would be a Stream of natural numbers: A Stream is a composable enumerable that is computed lazily Stream.iterate(0, fn x -> x + 1 end) The gory details of how this is implemented under the hood is the topic for another day. You rarely need to get that low level — the Stream library has a wealth of functions similar to that you can use as building blocks to construct your own Streams. And several Elixir library functions natively return streams: will turn a file (or other source) into a Stream allowing you to process very large files without having to read them into memory first in their entirety. The lazy aspect of Streams means that the minimum amount of the Stream is computed that’s needed to generate your answer. Enum IO.stream Imagine we want to find the 100th natural number that is palindromic and also divisible by 3: 1 
|> Stream.iterate(fn n -> n + 1 end) 
|> Stream.filter(fn n -> s = Integer.to_string(n); s == String.reverse(s) end)
|> Stream.filter(fn n -> rem(n, 3) == 0 end)
|> Enum.at(99) What I find elegant (and efficient) about this solution, is that no number beyond the answer, 20202, will be computed. Neat! If you’d like to dive in deeper, there’s a great Getting Started section on on the Elixir language website. The is also chock full of instructive examples. Enumerables and Streams Elixir API documentation Elasticsearch Now onto . The first thing to know is that After scrutinizing the documentation, it became clear that the original method that popped up on my Stack Overflow ( ) had been deprecated and the new approved method was to use . The process is: Elasticsearch ES caps the size of the result set at 10k items. Scroll API PointInTime IDs PIT IDs essentially give you a snapshot of your index at one point in time so that you get a consistent view of your search results across multiple queries. acquire a PIT for your index perform your queries using the PIT (without the index — it’s now implicit in the PID) delete the PIT That last step is crucial — PITs are extremely resource hungry so it is vital that you delete them as soon as you’re done. I borked our staging ES in my early experiments with PITs because I wasn’t deleting them and the expiration parameter seemed to have no effect. The other thing we’d like to do is abstract away the underlying PIT and paging mechanism for the end user so we provide an interface that just takes an ES query and generates a Stream of hits. Let’s start by creating and deleting the PID. is just a matter of requesting a PID associated with the index that we’re interested in running queries against. Creating the PID @spec create(ES.index_t()) :: {:ok, pit_t()} | {:error, ES.response_t()}
def create(index) do
  :post
  |> HTTP.request(url(index), HTTP.no_payload(), HTTP.json_encoding())
  |> HTTP.on_200(& &1["id"])
end is a simple HTTP delete except that the PIDs are huge so you need to supply them in the body and not as URL params. Some HTTP libraries won’t let you do this. The HTTP spec is a little muddy on the matter and last I checked on Stack Overflow there was a healthy debate on the subject. This is why we use HTTPoison — it allows payloads on DELETE requests. Deleting a PID @spec delete(pit_t()) :: :ok | {:error, HTTPoison.AsyncResponse | HTTPoison.MaybeRedirect | HTTPoison.Response}
def delete(pit) do
  url = HTTP.generate_url("_pit", ES.no_index(), HTTP.no_options())

  with {:ok, payload} <- Poison.encode(%{"id" => pit}),
        %HTTPoison.Response{status_code: 200} <-
          HTTPoison.request!(:delete, url, payload, HTTP.headers(HTTP.json_encoding())) do
    :ok
  else
    error -> {:error, error}
  end
end Our basic query is enhanced with three extra parameters: Now that the PIDs are sorted out, our next order of business is figuring out how to leverage them in our queries. size of 10,000 (the maximum allowed by elastic search) pit, a hash that contains %{id: <id>, keep_alive: “1m”} sort, a hash that contains %{_shard_doc: “desc”} (For sort, you need to provide something and is baked into every Elastic search index so that’s nice and portable.) _shard_doc @spec initial(PIT.pit_t(), ES.query_t()) :: ES.query_t()
def initial(pit, query) do
  %{
    query: query,
    pit: %{id: pit, keep_alive: PIT.expiration()},
    size: ES.max_results(),
    sort: %{"_shard_doc" => "desc"}
  }
end The trick here is that we feed into the next query the value of our sort field from the last hit of the previous result, like so: With our basic query down, we can focus on how to generate the sequence of queries that will extract all the items that satisfy our search criteria. @spec update(ES.query_t(), ES.response_t()) :: ES.query_t()
def update(query, %Req.Response{body: %{"hits" => %{"hits" => hits}}, status: 200}) do
  Map.put(query, :search_after, List.last(hits)["sort"])
end Armed with these two query-generating functions, we can code up that will pull all the results out of our desired index: Stream.iterator @spec streamer(ES.query_t()) :: {:ok, Enumerable.t()} | {:error, any}
def streamer(initial) do
  {:ok,
    initial
    |> search()
    |> Stream.iterate(fn previous -> search(Query.update(initial, previous)) end)
   }
rescue
  error -> {:error, error}
end We need a high-level function that takes the results from this function and produces a Stream of hits. But there’s a wrinkle — we have to delete the PIT when we are done with the Stream. But how will we know? The solution is to pass in a consumer that consumes the Stream and then we delete the PIT afterward, like so: @spec stream_many(ES.query_t(), ES.index_t(), ES.consumer_t(), non_neg_integer()) :: {:error, any()} | {:ok, Enumerable.t()}
def stream_many(query, index, consumer, count) do
  case PIT.create(index) do
    {:ok, pit_id} ->
      try do
        case pit_id |> Query.initial(query) |> streamer() do
          {:ok, stream} ->
            stream
            |> Stream.flat_map(& &1.body["hits"]["hits"])
            |> Stream.take(count)
            |> then(consumer)
          error -> error
        end
      after
        PIT.delete(pit_id)
      end
    error -> error
  end
rescue
  error -> {:error, error}
end But if our query returns less than 10k results, we don’t need the whole PID/sort/search_after machinery — one query will do the trick. @spec stream_one(ES.query_t(), ES.index_t(), ES.consumer_t()) :: {:ok, any()} | {:error, any()}
def stream_one(query, index, consumer) do
  query
  |> search(index)
  |> HTTP.on_200(fn body -> consumer.(body["hits"]["hits"]) end)
end Now we need a top-level function that query the size of the result and chooses between or depending on the value, like so: stream_one stream_many @spec stream(ES.query_t(), ES.index_t(), ES.consumer_t()) :: {:ok, any()} | {:error, any()}
def stream(query, index, consumer) do
  case count(query, index) do
    {:ok, count} ->
      if count > ES.max_results() do
        stream_many(query, index, consumer, count)
      else
        stream_one(query, index, consumer)
      end
    error ->
      error
  end
end AWS S3 The library we use to access is It handles all the low-level details for us but it does have one requirement: . Amazon S3 ex_aws_s3. the input data stream must be chunks of at least 4 Mbytes To accomplish this, we use the Stream function . This takes four inputs: chunk_while the enumerable to be chunked the initial value of the accumulator the step function a finalizer function Our first decision is, what should the accumulator look like that gets passed forward by the step function? It obviously should contain the list of the items in the current chunk. But more subtly, it should also contain the size of the current chunk so we don’t waste resources recomputing it each time. So that gives us a two-element tuple containing a list and an integer. Next, we turn our attention to the function. It should check if the current size is greater than or equal to the desired chunk size. If it is, we should take the list of items from the accumulator and convert them into a chunk using ; if not, we should add it to the current chunk (and update the size) using step convert add_chunk. What should do? Just push the item onto the front of a list and increase the value of by the current chunk’s size. add_chunk size The behaviour of depends on whether we care about the order of the items in the chunk being preserved in the output because the items in the list will be in reverse order so need to be reversed. But if we don’t care, we can skip that transformation. Putting this all together gives us: convert @spec chunker(Enumerable.t(), non_neg_integer(), boolean(), (any() -> non_neg_integer()), (Enumerable.t() -> any())) :: Enumerable.t()
def chunker(
        chunks,
        chunk_size,
        ordered \\ true,
        sizer \\ &String.length/1,
        joiner \\ &Enum.join/1
      ) do
  zero = {0, []}
  convert =
    if ordered do
      fn chunks -> chunks |> Enum.reverse() |> then(joiner) end
    else
      joiner
    end

  final = fn {_, chunks} -> {:cont, convert.(chunks), zero} end
  add_chunk = fn {size, chunks}, chunk -> {size + sizer.(chunk), [chunk | chunks]} end

  step = fn chunk, accum = {size, chunks} ->
    if size >= chunk_size do
      {:cont, convert.(chunks), add_chunk.(zero, chunk)}
    else
      {:cont, add_chunk.(accum, chunk)}
    end
  end

  Stream.chunk_while(chunks, zero, step, final)
end We can now combine all the components we have written to produce a simple controller action that requests an output stream of hits from our Elasticsearch section into 4M chunks that will satisfy the requirement of the ExAWS.S3 module: defmodule Controller do
  @four_megabytes 4*1024*1024

  def find_all(params) do
    query(params) |> ElasticSearch.stream(params.index, &consumer(params, &1))
  end
  
  def query(params) do
    %{term: %{params.query_term => params.query_value}}
  end
  
  def consumer(params, stream) do
    stream
    |> Stream.map(&Poison.encode!(&1["_source"]))
    |> chunker(@four_megabytes, false)
    |> S3.upload(params.bucket, params.filename)
    |> ExAws.request()
  end
end And that wraps up our exploration of Elixir streams, Elasticsearch, and AWS S3. I hope you enjoyed it! By Lob Staff Engineer, Guy Argo.

A Look at Using Elixir Streams, Elasticsearch & AWS S3

One very important thing in the software development process that is often overlooked in the early stages of a project is API documentation. Very often there are situations when developers do not describe the current API services due to tight deadlines or insufficient motivation, they plan to take up this task in the future. As a result, as the project grows, a large number of methods, parameters, and data formats appear, the description of which requires a huge amount of time and effort. This problem is especially critical when the development team is divided into small teams, such as front-end and back-end developers. In this case, without documentation, many problems can arise during the implementation process and a lot of time is spent on meetings and discussions of already implemented methods. Knowledge is passed orally from developer to developer. One of the solutions to this problem is frameworks for the automatic generation of documentation. For example, web applications often use swagger or its derivatives to describe REST APIs. This allows you to document the interfaces by which the front-end and back-end services interact. Swagger is a really good tool, but the interaction between different parts of the system is not limited to the REST API. In the case of dividing the project into microservices and using the Event-driven architecture, the interaction between services is built using events shared through the message broker. And each microservice is handled by a separate development team. It's reasonable to consider automatically documenting the events that are exchanged between services, just as swagger describes the REST API. In this article, we will look at how to enable automatic event documentation for services implemented using Spring Boot with interaction through Apache Kafka. Apache Kafka is a distributed streaming platform that allows you to process trillions of events per day. Kafka guarantees low latency, and high throughput provides fault-tolerant publish/subscribe pipelines and allows you to process event streams. To solve our problem, there is AsyncApi. AsyncAPI is an open source initiative that seeks to improve the current state of Event-Driven Architectures (EDA). AsyncApi has several Java tools that allow you to generate documentation from code. Springwolf is my choice because it provides a UI similar to springfox. Gradle dependencies To get started, we need to add dependencies to gradle.build implementation 'org.springframework.cloud:spring-cloud-starter-stream-kafka:3.2.2'
implementation 'javax.json:javax.json-api:1.1.4'
implementation 'org.glassfish:javax.json:1.1.4'

compileOnly 'org.projectlombok:lombok'

// Provides the documentation API
implementation 'io.github.springwolf:springwolf-kafka:0.6.1'
runtimeOnly 'io.github.springwolf:springwolf-ui:0.4.0' Description of events First, we need to define the events that we will send or expect from the topic. I will create an abstract DomainEvent class that will be extended with specific events and define serialization and deserialization rules in it using Jackson annotations. @NoArgsConstructor(access = AccessLevel.PROTECTED)
@AllArgsConstructor
@Getter
@Setter(AccessLevel.PROTECTED)
@EqualsAndHashCode
@JsonTypeInfo(
        use = JsonTypeInfo.Id.NAME,
        include = JsonTypeInfo.As.EXISTING_PROPERTY,
        property = "type",
        visible = true,
        defaultImpl = EntityChangedEvent.class
)
@JsonSubTypes(value = {
        @JsonSubTypes.Type(name = EntityChangedEvent.type, value = EntityChangedEvent.class),
        @JsonSubTypes.Type(name = EntityDeletedEvent.type, value = EntityDeletedEvent.class),
        @JsonSubTypes.Type(value = EntityChangedEvent.class)
})
@JsonIgnoreProperties(ignoreUnknown = true)
public abstract class DomainEvent {

    private String id;

    @JsonSerialize(using = LocalDateTimeSerializer.class)
    @JsonDeserialize(using = LocalDateTimeDeserializer.class)
    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss")
    private LocalDateTime occuredOn = LocalDateTime.now();

    public abstract String getType();

} Change event class. @NoArgsConstructor(access = AccessLevel.PROTECTED)
@Getter
@Setter(AccessLevel.PRIVATE)
@EqualsAndHashCode(callSuper = true)
@JsonInclude(JsonInclude.Include.NON_NULL)
public class EntityChangedEvent extends DomainEvent {
    public static final String type = "ENTITY_CHANGED_EVENT";

    private String title;
    private String description;
    private String code;
    @NonNull
    private String entityId;  

    @Builder
    public EntityChangedEvent(
            String id,
            @NonNull String entityId,
            @NonNull String code, 
            String title,
            String description
    ) {
        super(id, LocalDateTime.now());

        if(StringUtils.isAllBlank(title, description)) {
            throw new IllegalStateException("changes is none");
        }

        this.entityId = entityId;
        this.code = code;

        this.title = title;
        this.description = description;
    }

    @Override
    public String getType() {
        return type;
    }
} Delete event class @NoArgsConstructor(access = AccessLevel.PROTECTED)
@Getter
@Setter(AccessLevel.PRIVATE)
@EqualsAndHashCode(callSuper = true)
@JsonInclude(JsonInclude.Include.NON_NULL)
public class EntityDeletedEvent extends DomainEvent {
    public static final String type = "ENTITY_DELETED_EVENT";
    @NonNull
    private String entityId;

    @Builder
    public EntityDeletedEvent(
            String id,
            @NonNull String valueId
    ) {
        super(id, LocalDateTime.now());
        this.valueId = valueId;
    }

    @Override
    public String getType() {
        return type;
    }
} Configuring Consumers and KafkaListeners After describing the event classes, we can move on to the listener configuration. To do this, we will get the configurations class KafkaConsumerConfig, in which we will initiate Beans for the ConcurrentKafkaListenerContainerFactory. We will also define the component KafkaListeners for our factory. If there are several factories, for example, if we have several families of events that need to be divided, then for each it is necessary to create a similar class. /**
Consumer configuration class for initiate factories for event families
*/
@Configuration
public class KafkaConsumerConfig {

    private final String SERVER;
    private final String SERVER_PORT;
    private final String GROUP;

    public KafkaConsumerConfig(
            @Value("${spring.cloud.stream.bindings.service-out-0.group}") String group,
            @Value("${spring.cloud.stream.kafka.binder.brokers}") String server,
            @Value("${spring.cloud.stream.kafka.binder.defaultBrokerPort}") String port
    ) {
        this.GROUP = group;
        this.SERVER = server;
        this.SERVER_PORT = port;
    }

/*
each key will be serialized to String and each event will be serialized to JSON
*/
    private <T> ConsumerFactory<String, T> typeConsumerFactory(Class<T> clazz) {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, String.format("%s:%s", SERVER, SERVER_PORT));
        props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP);

        return new DefaultKafkaConsumerFactory<>(
                props,
                new StringDeserializer(),
                new JsonDeserializer<>(clazz));
    }

    private <T> ConcurrentKafkaListenerContainerFactory<String, T> initFactory(Class<T> clazz) {
        ConcurrentKafkaListenerContainerFactory<String, T> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(typeConsumerFactory(clazz));
        return factory;
    }

/*
init bean factory for abstract DomainEvent. This factory accept all subclasses of DomainEvent
*/
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, DomainEvent> domainEventKafkaListenerContainerFactory() {
        return initFactory(DomainEvent.class);
    }

} KafkaListener class must be define for each factory. @Component
@Log
@KafkaListener(
        topics = "topic-test",
        groupId="group-test",
        containerFactory="domainEventKafkaListenerContainerFactory")
public class KafkaListeners {

    @KafkaHandler
    void entityChangedKafkaListener(@Payload EntityChangedEvent message) {
        log.info(String.format("KafkaHandler[EntityChanged] %s %s %s", message.getType(), message.getEntityId(), message.getId()));
    }

    @KafkaHandler
    void entityDeletedKafkaListener(@Payload EntityDeletedEvent message) {
        log.info(String.format("KafkaHandler[EntityDeleted] %s %s %s", message.getType(), message.getEntityId(), message.getId()));
    }
} springwolf configuration Finally, we can move on to the springwolf configuration. We need to set up information about the application, producers, and consumers. Consumers are configured automatically based on the KafkaListener and KafkaHandler annotations, however, producers must be enumerated manually. The whole config will look something like this. @Configuration
@EnableAsyncApi
public class AsyncApiConfiguration {
    private final String SERVER;
    private final String SERVER_PORT;
    private final String CONSUMERS_BASE_PACKAGE = "consumers.package";
    private final String CHANNEL_NAME;


    public AsyncApiConfiguration(
            @Value("${spring.cloud.stream.bindings.service-out-0.destination}") String channelName,
            @Value("${spring.cloud.stream.kafka.binder.brokers}") String server,
            @Value("${spring.cloud.stream.kafka.binder.defaultBrokerPort}") String port
    ) {
        this.CHANNEL_NAME = channelName;
        this.SERVER = server;
        this.SERVER_PORT = port;
    }

    @Bean
    public AsyncApiDocket asyncApiDocket() {
        //application info
        Info info = Info.builder()
                .version("api version")
                .title("service name")
                .description("description")
                .build();

        //kafka server info
        Server kafkaServer = Server.builder()
                .protocol("kafka")
                .url(String.format("%s:%s", SERVER, SERVER_PORT))
                .build();

        //init producer builder
        KafkaProducerData.ProducerDataBuilder producerBuilder = ProducerData.builder()
                .channelName(CHANNEL_NAME)
                .channelBinding(Map.of("kafka", new KafkaChannelBinding()))
                .operationBinding(Map.of("kafka", new KafkaOperationBinding()));

        //list of event types for documentation
        List<Class<?>> events = List.of(
                EntityChangedEvent.class,
                EntityDeletedEvent.class
        );

        //generate producers
        List<ProducerData> producers = events.stream()
                .map(c -> producerBuilder
                        .payloadType(c)
                        .build()
                ).collect(Collectors.toList());

        //build AsyncApiDocket
        return AsyncApiDocket.builder()
                .basePackage(CONSUMERS_BASE_PACKAGE) //package for search KafkaListeners
                .info(info) //application info
                .producers(producers) //producers
                .server("kafka", kafkaServer) //kafka srever info
                .build();
    }
} The interface will be available at . Documentation in JSON format can be found at /springwolf/asyncapi-ui.html /springwolf/docs Conclusion In this article, we looked at how you can set up auto-generation of documentation for an Event-Driven architecture. Do not forget to take care of this at the start of your project to reduce costs in the future. Links https://www.asyncapi.com/docs/tools https://github.com/springwolf/springwolf-core https://springwolf.github.io/docs/quickstart/

Auto-generation of Documentation for Event-driven Architecture

Flutter has been booming worldwide from the past few years. While there are many popular mobile app development technologies out there, Flutter has managed to leave its mark in the mobile application development world. In this article, we’ve curated the best Flutter app templates available on the market as of July 2020. Flutter can be the framework model for every app out there. This app development framework has taken the world by storm with its simplicity, widgets, and pixel-by-pixel based mobile UI development. Simply, the apps developed using Flutter are beautiful and feature-rich. Why Flutter? Flutter makes the app development simpler by applying a Widgets mechanism. Basically, we can create the application interface by simply stacking the Widget tree with different Widgets. The flow of Widgets across the code determines the flow of UI elements on the mobile screen. It makes use of the Dart programming language at its core. Moreover, the learning curve of Flutter and Dart programming language is not that steep. It easy in comparison with other application development techs. It delivers a cross-platform application development functionality. This means the same coding implementation can be applied to create both Android and iOS-based applications. It offers many libraries out of the box. Not to mention that a plethora of libraries are out there which a Flutter developer can use to easily implement wonderful mobile app features. The templates can act as a learning material for any Flutter developer especially the beginners. With this in mind, we present you the 25 best open-source, as well as premium, Flutter app templates out there for you to explore. 1. Best Flutter App UI Templates Here is a GitHub repository offering the fully functional Flutter templates. The templates are open-source. Hence, everyone can use it freely. The templates include the design implementation for hotel booking, Fitness, design course, and custom drawer. These templates can help any Flutter developer especially the beginners to learn and get started with their own custom app. You can choose between the hotel booking UI or Fitness app UI. The features like card list view with beautiful UI and designs are intuitive and easy to use. One looking for booking apps, fitness apps or course app in Flutter environment should definitely try these templates out. 2. Flutter Chat App This template offers a complete chat app feature written entirely in Flutter and Dart. The app theme is integrated with Firebase backend. It delivers standard chat app features like real-time conversations, push notifications as well as group chat and video message functionality. It provides 30+ screens with beautiful and subtle chat app UI designs. Our favorite Flutter app theme includes the integration of camera, online status indicator, profile screen options, social media logins, custom loaders, etc. It also houses the modern Android OS functionality such as dark mode. 3 . Flutter Dating App Template This is one of the best premium Flutter app templates out there which was closely based on the Tinder app. It offers a fully functional dating app template with Firebase server backend integration. This template is entirely written in Flutter. The UI flow and designs are inspired by the Tinder UI kit. This template supports the match-making algorithm along with notifications and real-time messaging. The design implementation supports both Android and iOS platforms. Inspired by the Tinder app, this app also incorporates the swipe mechanism along with date recommendations. The features like matches, real-time conversations, dating/personal profiles, localizations are a must-have in any dating app. Thus, this template takes an extra effort to provide these features with optimized performance. The new Android features like dark-mode support is also included. 4. Flutter WordPress App Template This app template is completely written in Flutter and Dart, providing native experience and great performance. It offers optimized collection views, cross-platform support, and modular code. The UI of this template exhibits beautiful newsreader designs. The mobile template is compatible with the integration of any WordPress website. This is one of the best Flutter app templates available on the market. Users can specify the WordPress URL and the app will automatically fetch the article. Moreover, the UI implementations include an automatic slide menu with WordPress categories, articles pagination / batching, for great performance, news list, and pull to refresh functionality. Thus, this template also houses real-time updates which automatically fetches the latest articles on the list. 5. Grocery, Food, Pharmacy, Store Delivery Mobile App with Admin Panel The template is a premium tier template available in the Codecanyon marketplace for the price of $79. This mobile app template delivers market directory management system using a Flutter development environment. It offers Google Maps for showing restaurants on the maps with direction to them. 6. News Buzz New Buzz repository available freely in GitHub delivers a News App template created in Flutter. It uses News API for fetching real-time data and Firebase as the backend and authenticator. The features includes custom news feed based on selected sources, saving articles, explore news based on categories, search for topics, and sharing articles. Hence, this may be the best solution for creating your very own Flutter news app. 7. Flutter Music Player Flutter Music Player is the first open-source Flutter based material design music player with audio plugin to play local music files. The app demo is available in . As of now, the template features only support the Android platform with options for iOS features in the future. The feature integrated for Android platform include fetch local songs (Inbuilt Permission), beautiful UI with multiple themes, animation, full-fledged example, play (local files), stop, pause, seek, shuffle, album art, incomplete, onDuration / onCurrentPosition functionalities. Play Store Moreover, this project also has a plugin to fetch songs and use an audio player that can be used in any of your projects. Thus, this template might be the best solution if you are keen on building your very own music player using Flutter. 8. Fluttergram As the name suggests, the Fluttergram template is an open-source template that provides a fully-functional Instagram clone written in Flutter using Firebase / Firestore. The features are similar to those of the Instagram app. The template offers custom photo feed based on who you follow (using firebase cloud functions), post photo from camera or gallery, like posts, comment on posts, view all comments on a post, search for users, and much more. Thus, if you are looking to implement your own social media app like Instagram, then this template may be the way to get it all started. 9. BookSearch BookSearch app template is an open-source app template related to book collecting and reading. The implementation is totally based on Flutter and Dart. It offers four views for book listing, collecting, searching, and reading. Moreover, there is the feature of searching as well as bookmarking your favorite books. If you are looking to create your very own books aggregator and reading app then this app template may be the best option for you. 10. Beer Me Up This app template is particularly unusual as it has the functionality of logging the beers you drink. However, this app template may be a good source for learning the new Flutter UI, as well as customize the UI based on your needs. The template is a personal beer logging app that allows you to enter every beer you have. Hence, it makes sure to keep a log of them so that you can remember them forever. 11. Food Delivery Flutter + PHP Laravel Admin Panel Here is another premium tier Flutter app template available in the Codecanyon market place for 79 USD. The app template offers multiple Restaurants or Restaurants directory Management System. There is the feature of Google Maps for showing restaurants on the map with directions to them. It delivers entities & Models that are ready to use, clean & organized Dart Language code using MVC Pattern, easy to restyle, and theming by on your branding sites, as well as cash on delivery and payment on pickup methods. 14. Trace Trace is an open-source full-featured, modern crypto portfolio & market explorer. The features include Portfolio Overview, Aggregate Market Explorer, Portfolio Timeline, Transaction History, Portfolio Breakdown, and much more. 15. Taskist Taskist is an open-source to-do list app written completely in Flutter. The UI is simple, beautiful, and intuitive. It offers a task management system with the ability to add multiple tasks in the list format. The app uses Firebase to store the list. The UI features is based on swiping mechanism where the list can swipe right for more. There is an option of striking out the completed tasks. Moreover, it also features the ability to store and maintain tasks based on different days. 16. Fluxstore Pro – Flutter E-commerce Full App Fluxstore is a premium tier e-commerce app inspired by the Flutter framework available in the Codecanyon marketplace for $79. With the aim to reduce hours of business spent on designing, developing, and testing a mobile app, it comes as a complete solution for optimizing to deliver your app to the market with high productivity and cost-efficiency. Moreover, it offers features that satisfy all of the business requirements including e-commerce functionalities, impressive UX design, and smooth performance on both iOS and Android devices. 17. Trinity Orientation 2018 This is an app created for Trinity College at the University of Toronto. The app acts as a companion for all first-year students during orientation week. The features include college sitemap, map functionality, chat functionality, events list, gallery, etc. This app may be useful for creating similar apps for other colleges. Nonetheless, it can be used as the learning mechanism for Flutter development. 18. Deer Here is another to-do list based app template which is available freely in GitHub. This template is a minimalist Todo Planner app built around the idea of efficiency and clean aesthetic. It makes use of the BLoC (Business Logic Component) pattern to manage the app state. It showcases 4 screens. The first one is the list entry screen where you enter the to-do task, then the list screen to view the to-do task list, the screen for task list detail, and the screen to edit the to-do task. Thus, this template can be the right choice to develop your very own to-do list app using Flutter in the BLoC pattern. 19. TailorMade TailorMade is an open-source daily routine management app template made especially for Fashion designers. It started out as an experiment with Flutter, ReBLoC, and but later turned into a valuable tool for managing a Fashion designer’s daily routine. Firebase Cloud Functions Moreover, the template is simple, clean, and overall has a very smooth intuitive feel. This template offers 20+ screens which are all fully functional. Hence, it is highly suggested to check this template out as you can have new features to learn in Flutter and also develop your very own routine management app with some tweaks. 20. instory instory is an open-source app available in Github free of cost. This mobile template delivers a Flutter app with beautiful UI features with 3 screens. The major objective of this app template is to download Instagram stories. Moreover, the app offers the search functionality to search for a particular profile story and download them. 21. StreamApp – Streaming Movies, TV Series and Live TV Stream app is another premium tier app available in the CodeCanyon store for $29. It delivers features for streaming movies, TV series, and live TV. This app template has support for formats on external servers, local or AWS S3 storage. It saves a lot of effort for the administrator with features such as remote configuration, auto-import of information from TheMovieDB, integration with AWS API, Firebase, AdMob, and more. Moreover, the admin panel built using Vue.js and backed by Laravel guarantees you the full power of new technologies. 22. Reply  Flutter This Flutter app template delivers a pixel-perfect implementation of the in . is an email app that makes use of Material Design components and Material Theming to implement an on-brand communication experience. Reply Material Design Case Study Flutter Reply Material Design Case Study It mainly focuses on UI elements of Flutter such as animation, layouts, and transitions. This template aims to showcase how powerful Flutter has become as a UI framework. Hence, this template can be used as a learning mechanism for Flutter app development. 23. Enigma Enigma is an open-source one-to-one chat app developed entirely in Flutter. The backend is based on Firebase implementation. This app template delivers real-time one-to-one chatting experience with minimal chatting features. Since this is a Firebase dependent project, it uses Firebase Phone Authentication, Cloud Firestore, Firebase Storage, and Firebase In-App Messaging. Moreover, the chat interfaces are passcode protected. This template can be a medium to learn how to develop a Flutter chatting app using Firebase features. Hence, you can also develop your very own Flutter chatting app using this template as a starter package. 24. Flutter Pokédex How can we forget one of the legendary anime Pokémon? There has been a number of games based on it as well. Those who watch Pokémon or play their games simply cannot ignore the vital use of the Pokédex. Well, this template offers the functionality of a Pokédex and is completely built using Flutter. The app hosts the Pokémon list along with their data. Moreover, what makes this template great is the Widget combination, animations, and charts used in it. This can be a great app to learn the use of animations and different UI components in Flutter. Even if you are not a fan of Pokémon, you should still check this template out. Conclusion With this plethora of Flutter app templates available out there, one can surely develop the app they are looking for. Most app templates are open-source which means they are free to use as the starter template for creating your very own Flutter application. Some premium tier templates are available which provide powerful features that you can mildly customize based on your needs and deploy to the market. The best Flutter app templates can be the breeding ground for many application startups and solutions for companies. Thus, it is highly suggested to check each one of these flutter app templates for a better understanding of the Flutter app market and its use cases. Did we miss your favorite Flutter app template? Let us know in the comments! Previously published at https://www.instaflutter.com/design/flutter-app-templates-examples-best-25-in-2020/

Top 25 Flutter Mobile App Templates in 2020

Adding. Changing. Hacking flavours. After 3 years of developing, I’ve begun publicly marketing . Streamplate Over time it’s shifted in design, scope and intent, and now stands as an operating-like-system for consumers and merchants alike to manage their eating and drinking needs. At its core, Streamplate offers a personalised assortment of merchants with a sorted listing of menu-items to cater for each user’s needs. While the first iteration of the Alpha app has been completed, there’s still much to be done. Presently, each computation of the merchant listings takes into account approximately a few thousand-variables. What I’ve realised over the past few months is that while having such a high-dimensional space isn’t anything unique, the frequency of these calculations is. Eating and drinking isn’t based on “universal truths” which sets it apart from the paradigms that guide Google’s search engine or Spotifys’ playlists curations. What someone wants to order is a function of their perspective in that moment in time. When we break down what a “perspective” is in factor-analysis, a high-level structure emerges: Perception => State of values, Consciousness => Stream of perceptions, Levels of consciousness => Attentiveness to rates of changing intensities of consciousness. A key-insight here is how underlying any listing is a measurement of “values”, hence the need for a broad-spectrum of values and hence an increasingly-growing dimensional-space. What this means is that Streamplate needs to consistently be measuring the pulse of a user’s perception so when they do call for a listing, we can be sure to accurately match their needs. Think of how varying your tastes can be, particularly under duresses like unforgiving summer heat or compelling social influence. Reaching this understanding required a progressive set of design iterations, each time further investigating the guiding principles affecting our users’ decisions. As I continue working on the back-end code to implement these understandings, I’d like to briefly share some moments in the design process. Jump to the front of the queue for a year by signing up for Streamplate’s ! Limited Release

Designing Streamplate

Real-Time tricks: Harnessing Kafka Streams for Seamless Data Tasks

Python Bokeh is one of the best Python packages for data visualization. Today, we are going to see some Python Bokeh Examples. I have also provided the Python Bokeh project source code on GitHub. Learn this easy visualization tool and add it to your Python stack. What is Python Bokeh? Python is a data visualization tool, or we can also say Python Bokeh is used to plot various types of graphs. There are various other graph plotting libraries like matplotlib, but Python Bokeh graphs are dynamic in nature, which means you can interact with the generated graph. See the below examples… Bokeh Installation 💻: Python Bokeh can be easily installed using PIP. You can install the Python Bokeh easily by running the command: pip install bokeh Now everything is ready, let’s go through the examples 🏃‍♂️… 1. LinePlot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="{LinePlot Python Bokeh Example")
p.line(x, y, line_width=2)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 2. Scatter Plot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Scatter Plot Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=10, color="navy", alpha=0.5)
output_notebook()
show(p) 3. Bar Chart from bokeh.plotting import figure, show, output_notebook
categories = ["A", "B", "C", "D", "E"]
values = [10, 15, 8, 12, 6]
p = figure(x_range=categories, title="Bar Chart Python Bokeh Example by PratikPathak.com")
p.vbar(x=categories, top=values, width=0.9)
output_notebook()
show(p) 4. Histogram from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.normal(0, 1, 1000)
p = figure(title="Histogram Python Bokeh Example by PratikPathak.com")
p.hist(data, bins=30, color="navy", alpha=0.5)
output_notebook()
show(p) 5. Pie Chart from bokeh.plotting import figure, show, output_notebook
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
p = figure(title="Pie Chart Python Bokeh Example by PratikPathak.com")
p.wedge(x=0, y=0, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_label=labels)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 6. Time Series Plot from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Time Series Plot Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 7. Linked Brushing from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
source = ColumnDataSource(data=dict(x1=x1, y1=y1, x2=x2, y2=y2))
p1 = figure(title="Scatter Plot 1")
p1.circle('x1', 'y1', source=source)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle('x2', 'y2', source=source)
output_notebook()
show(p1, p2) 8. Hover Tooltips from bokeh.plotting import figure, show, output_notebook
from bokeh.models import HoverTool
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Hover Tooltips Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) 9. Annotations from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Arrow, VectorRenderer, Label
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Annotations Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
arrow = Arrow(x_start=2, y_start=4, x_end=3, y_end=5, line_width=2, line_color="red")
label = Label(x=3, y=6, text="This is a label", render_mode='css', border_line_color='black', border_line_alpha=1.0, background_fill_color='white', background_fill_alpha=0.5)
p.add_layout(arrow)
p.add_layout(label)
output_notebook()
show(p) 10. Custom Glyphs from bokeh.plotting import figure, show, output_notebook
from bokeh.models.glyphs import Asterisk
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Custom Glyphs Python Bokeh Example by PratikPathak.com")
p.add_glyph(x, y, Asterisk(size=20, line_color="red", fill_color="yellow"))
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 11. Gridlines and Axes from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Gridlines and Axes Python Bokeh Example by PratikPathak.com", x_range=(0, 6), y_range=(0, 8))
p.grid.grid_line_color = "grey"
p.grid.grid_line_dash = [6, 4]
p.xaxis.axis_label = "X-axis"
p.yaxis.axis_label = "Y-axis"
p.line(x, y, line_width=2)
output_notebook()
show(p) 12. Legend from bokeh.plotting import figure, show, output_notebook
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p = figure(title="Legend Python Bokeh Example by PratikPathak.com")
p.line(x1, y1, line_width=2, color="red", legend_label="Line 1")
p.line(x2, y2, line_width=2, color="blue", legend_label="Line 2")
p.legend.location = "top_left"
output_notebook()
show(p) 13. Categorical Plots from bokeh.plotting import figure, show, output_notebook
fruits = ["Apples", "Pears", "Nectarines", "Plums", "Grapes", "Strawberries"]
counts = [5, 3, 4, 2, 4, 6]
p = figure(x_range=fruits, title="Categorical Plots Python Bokeh Example by PratikPathak.com")
p.vbar(x=fruits, top=counts, width=0.9)
p.xaxis.axis_label = "Fruit"
p.yaxis.axis_label = "Count"
p.xaxis.axis_label_text_font_size = "12pt"
p.yaxis.axis_label_text_font_size = "12pt"
output_notebook()
show(p) 14. Subplots from bokeh.plotting import figure, show, output_notebook
from bokeh.layouts import gridplot
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com")
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
grid = gridplot([[p1, p2]], plot_width=400, plot_height=400)
output_notebook()
show(grid) 15. Interactive Plots from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, BoxSelectTool
import numpy as np
x = np.random.random(1000)
y = np.random.random(1000)
source = ColumnDataSource(data=dict(x=x, y=y))
p = figure(title="Interactive Plots Python Bokeh Example by PratikPathak.com", tools="hover,box_select")
p.circle('x', 'y', source=source, size=3, color="navy", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 16. Linked Panning and Zooming from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com", x_range=Range1d(0, 6), y_range=Range1d(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 17. Linked Axis Ranges from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1", x_range=(0, 6), y_range=(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 18. DateTime Axis from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Datetime Axis Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 19. Heatmap from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.rand(10, 10)
p = figure(title="Heatmap Python Bokeh Example by PratikPathak.com")
p.image(image=[data], x=0, y=0, dw=10, dh=10, palette="Viridis256")
output_notebook()
show(p) 20. Streamgraph from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
import numpy as np
# Generate some sample data
x = np.linspace(0, 10, 100)
y1 = np.cumsum(np.random.randn(100))
y2 = np.cumsum(np.random.randn(100))
y3 = np.cumsum(np.random.randn(100))
y4 = np.cumsum(np.random.randn(100))
y5 = np.cumsum(np.random.randn(100))
source = ColumnDataSource(data=dict(x=x, y1=y1, y2=y2, y3=y3, y4=y4, y5=y5))
p = figure(title="Streamgraph Python Bokeh Example by PratikPathak.com")
p.area(x='x', y1='y1', y2='y2', y3='y3', y4='y4', y5='y5', source=source, line_color=None)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 21. Polar Plot from bokeh.plotting import figure, show, output_notebook
import numpy as np
r = np.linspace(0.1, 1, 10)
theta = np.linspace(0, 2*np.pi, 10)
p = figure(title="Polar Plot Python Bokeh Example by PratikPathak.com", plot_width=600, plot_height=600, x_range=(-1.2, 1.2), y_range=(-1.2, 1.2), toolbar_location=None)
p.polar(r, theta, color="navy", alpha=0.5, line_width=2)
output_notebook()
show(p) 22. Donut Chart from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
source = ColumnDataSource(data=dict(labels=labels, values=values))
p = figure(title="Donut Chart Python Bokeh Example by PratikPathak.com", x_range=labels)
p.wedge(x=0, y=1, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_field="labels")
p.wedge(x=0, y=1, inner_radius=0.3, outer_radius=0.4, start_angle=0.6, end_angle=2.6, color="white")
p.axis.visible = False
p.grid.grid_line_color = None
output_notebook()
show(p) 23. Treemap from bokeh.plotting import figure, show, output_notebook
from bokeh.models import TreemapRenderer, ColumnDataSource
import pandas as pd

# Sample data
data = pd.DataFrame({
    'label': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H'],
    'parent': ['', 'A', 'A', 'B', 'B', 'C', 'C', 'D'],
    'value': [10, 5, 4, 3, 2, 1, 1, 1]
})

source = ColumnDataSource(data)
p = figure(title="Treemap Python Bokeh Example by PratikPathak.com")
treemap = TreemapRenderer(
    source=source,
    label='label',
    parent='parent',
    value='value',
    color_discrete_sequence=["#e6550d", "#3182bd", "#31a354", "#756bb1", "#636363", "#9e9ac8", "#de2d26", "#fd8d3c"],
)
p.add_layout(treemap)
output_notebook()
show(p) If you get any error, make sure you have installed Python pandas by using the “pip install pandas” command 24. Network Plot from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, NodesAndLinkedEdges
import networkx as nx

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
edges = [('A', 'B'), ('A', 'C'), ('B', 'D'), ('B', 'E'), ('C', 'F')]
G = nx.Graph()
G.add_nodes_from(nodes)
G.add_edges_from(edges)

node_positions = nx.spring_layout(G)
node_x = [node_positions[node][0] for node in G.nodes()]
node_y = [node_positions[node][1] for node in G.nodes()]
node_color = ['red', 'green', 'blue', 'yellow', 'purple', 'orange']

source = ColumnDataSource(data=dict(
    x=node_x,
    y=node_y,
    color=node_color,
    node=nodes
))

p = figure(title="Network Plot Python Bokeh Example by PratikPathak.com", x_range=(-1.1, 1.1), y_range=(-1.1, 1.1))
p.add_tools(HoverTool(tooltips=[("Node", "@node")]))
p.circle('x', 'y', size=15, color='color', source=source)

edge_x = []
edge_y = []
for start, end in edges:
    x0, y0 = node_positions[start]
    x1, y1 = node_positions[end]
    edge_x.append(x0)
    edge_x.append(x1)
    edge_x.append(None)
    edge_y.append(y0)
    edge_y.append(y1)
    edge_y.append(None)

p.multi_line(edge_x, edge_y, color="black", alpha=0.5, line_width=1)

output_notebook()
show(p) 25. Sankey Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Sankey, ColumnDataSource

# Sample data
flows = [
    ('A', 'X', 5),
    ('B', 'X', 3),
    ('A', 'Y', 2),
    ('B', 'Y', 4),
    ('X', 'C', 4),
    ('X', 'D', 3),
    ('Y', 'C', 2),
    ('Y', 'D', 4)
]

source = ColumnDataSource(data=dict(
    source=[flow[0] for flow in flows],
    target=[flow[1] for flow in flows],
    value=[flow[2] for flow in flows]
))

p = figure(plot_width=800, plot_height=400, title="Sankey Diagram Python Bokeh Example by PratikPathak.com")
sankey = Sankey(source=source, label_text_color="white", label_text_font_size="8pt")
p.renderers.append(sankey)

output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 26. Chord Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, CategoricalColorMapper
import numpy as np

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
connections = [
    ('A', 'B', 0.5), ('A', 'C', 0.3), ('A', 'D', 0.2),
    ('B', 'C', 0.4), ('B', 'D', 0.3), ('B', 'E', 0.3),
    ('C', 'D', 0.2), ('C', 'E', 0.3), ('C', 'F', 0.5),
    ('D', 'E', 0.2), ('D', 'F', 0.3), ('E', 'F', 0.4)
]

source = ColumnDataSource(data=dict(
    start=np.array([con[0] for con in connections]),
    end=np.array([con[1] for con in connections]),
    value=np.array([con[2] for con in connections])
))

p = figure(title="Chord Diagram Python Bokeh Example by PratikPathak.com", plot_width=800, plot_height=800)
chord = p.chord(
    source=source, 
    start_angle=np.pi / 2,
    direction="clockwise",
    colors=CategoricalColorMapper(factors=nodes, palette="Spectral11")
)

output_notebook()
show(p) How to Contribute? Feel free to open a PR request on our GitHub repo. Steps to contribute: Fork the repo Make changes in the Forked repo and save Open a Pull Request That’s it 😄! 🌿 Contribute Conclusion In this Repo, I have shared 25+ Python Bokeh examples that can help you learn Python Bokeh. Feel free to contribute to our GitHub repo and keep it updated. Also published . here

STREAMKAP

streamkap.com

-- employees

Since n.d.

COMPANY RANKING

STREAMKAP

EVERGREEN INDEX #10304

Representing this company? Click here to claim and customize this page!

Streamkap WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

STREAMKAP

streamkap.com

-- employees

Since n.d.

COMPANY RANKING

STREAMKAP

EVERGREEN INDEX #10304

RELATED COMPANIESRANK

STORIES ABOUT STREAMKAP

Representing this company? Click here to claim and customize this page!

Streamkap WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!