Source is a data management layer built for the decentralized web, a fast-growing market expected to reach $40B by 2025

Source

How to Run Your Own Local LLMs: Updated for 2024 with 8 New Open Source Tools!

Python Bokeh is one of the best Python packages for data visualization. Today, we are going to see some Python Bokeh Examples. I have also provided the Python Bokeh project source code on GitHub. Learn this easy visualization tool and add it to your Python stack. What is Python Bokeh? Python is a data visualization tool, or we can also say Python Bokeh is used to plot various types of graphs. There are various other graph plotting libraries like matplotlib, but Python Bokeh graphs are dynamic in nature, which means you can interact with the generated graph. See the below examples… Bokeh Installation 💻: Python Bokeh can be easily installed using PIP. You can install the Python Bokeh easily by running the command: pip install bokeh Now everything is ready, let’s go through the examples 🏃‍♂️… 1. LinePlot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="{LinePlot Python Bokeh Example")
p.line(x, y, line_width=2)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 2. Scatter Plot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Scatter Plot Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=10, color="navy", alpha=0.5)
output_notebook()
show(p) 3. Bar Chart from bokeh.plotting import figure, show, output_notebook
categories = ["A", "B", "C", "D", "E"]
values = [10, 15, 8, 12, 6]
p = figure(x_range=categories, title="Bar Chart Python Bokeh Example by PratikPathak.com")
p.vbar(x=categories, top=values, width=0.9)
output_notebook()
show(p) 4. Histogram from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.normal(0, 1, 1000)
p = figure(title="Histogram Python Bokeh Example by PratikPathak.com")
p.hist(data, bins=30, color="navy", alpha=0.5)
output_notebook()
show(p) 5. Pie Chart from bokeh.plotting import figure, show, output_notebook
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
p = figure(title="Pie Chart Python Bokeh Example by PratikPathak.com")
p.wedge(x=0, y=0, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_label=labels)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 6. Time Series Plot from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Time Series Plot Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 7. Linked Brushing from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
source = ColumnDataSource(data=dict(x1=x1, y1=y1, x2=x2, y2=y2))
p1 = figure(title="Scatter Plot 1")
p1.circle('x1', 'y1', source=source)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle('x2', 'y2', source=source)
output_notebook()
show(p1, p2) 8. Hover Tooltips from bokeh.plotting import figure, show, output_notebook
from bokeh.models import HoverTool
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Hover Tooltips Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) 9. Annotations from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Arrow, VectorRenderer, Label
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Annotations Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
arrow = Arrow(x_start=2, y_start=4, x_end=3, y_end=5, line_width=2, line_color="red")
label = Label(x=3, y=6, text="This is a label", render_mode='css', border_line_color='black', border_line_alpha=1.0, background_fill_color='white', background_fill_alpha=0.5)
p.add_layout(arrow)
p.add_layout(label)
output_notebook()
show(p) 10. Custom Glyphs from bokeh.plotting import figure, show, output_notebook
from bokeh.models.glyphs import Asterisk
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Custom Glyphs Python Bokeh Example by PratikPathak.com")
p.add_glyph(x, y, Asterisk(size=20, line_color="red", fill_color="yellow"))
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 11. Gridlines and Axes from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Gridlines and Axes Python Bokeh Example by PratikPathak.com", x_range=(0, 6), y_range=(0, 8))
p.grid.grid_line_color = "grey"
p.grid.grid_line_dash = [6, 4]
p.xaxis.axis_label = "X-axis"
p.yaxis.axis_label = "Y-axis"
p.line(x, y, line_width=2)
output_notebook()
show(p) 12. Legend from bokeh.plotting import figure, show, output_notebook
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p = figure(title="Legend Python Bokeh Example by PratikPathak.com")
p.line(x1, y1, line_width=2, color="red", legend_label="Line 1")
p.line(x2, y2, line_width=2, color="blue", legend_label="Line 2")
p.legend.location = "top_left"
output_notebook()
show(p) 13. Categorical Plots from bokeh.plotting import figure, show, output_notebook
fruits = ["Apples", "Pears", "Nectarines", "Plums", "Grapes", "Strawberries"]
counts = [5, 3, 4, 2, 4, 6]
p = figure(x_range=fruits, title="Categorical Plots Python Bokeh Example by PratikPathak.com")
p.vbar(x=fruits, top=counts, width=0.9)
p.xaxis.axis_label = "Fruit"
p.yaxis.axis_label = "Count"
p.xaxis.axis_label_text_font_size = "12pt"
p.yaxis.axis_label_text_font_size = "12pt"
output_notebook()
show(p) 14. Subplots from bokeh.plotting import figure, show, output_notebook
from bokeh.layouts import gridplot
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com")
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
grid = gridplot([[p1, p2]], plot_width=400, plot_height=400)
output_notebook()
show(grid) 15. Interactive Plots from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, BoxSelectTool
import numpy as np
x = np.random.random(1000)
y = np.random.random(1000)
source = ColumnDataSource(data=dict(x=x, y=y))
p = figure(title="Interactive Plots Python Bokeh Example by PratikPathak.com", tools="hover,box_select")
p.circle('x', 'y', source=source, size=3, color="navy", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 16. Linked Panning and Zooming from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com", x_range=Range1d(0, 6), y_range=Range1d(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 17. Linked Axis Ranges from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1", x_range=(0, 6), y_range=(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 18. DateTime Axis from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Datetime Axis Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 19. Heatmap from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.rand(10, 10)
p = figure(title="Heatmap Python Bokeh Example by PratikPathak.com")
p.image(image=[data], x=0, y=0, dw=10, dh=10, palette="Viridis256")
output_notebook()
show(p) 20. Streamgraph from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
import numpy as np
# Generate some sample data
x = np.linspace(0, 10, 100)
y1 = np.cumsum(np.random.randn(100))
y2 = np.cumsum(np.random.randn(100))
y3 = np.cumsum(np.random.randn(100))
y4 = np.cumsum(np.random.randn(100))
y5 = np.cumsum(np.random.randn(100))
source = ColumnDataSource(data=dict(x=x, y1=y1, y2=y2, y3=y3, y4=y4, y5=y5))
p = figure(title="Streamgraph Python Bokeh Example by PratikPathak.com")
p.area(x='x', y1='y1', y2='y2', y3='y3', y4='y4', y5='y5', source=source, line_color=None)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 21. Polar Plot from bokeh.plotting import figure, show, output_notebook
import numpy as np
r = np.linspace(0.1, 1, 10)
theta = np.linspace(0, 2*np.pi, 10)
p = figure(title="Polar Plot Python Bokeh Example by PratikPathak.com", plot_width=600, plot_height=600, x_range=(-1.2, 1.2), y_range=(-1.2, 1.2), toolbar_location=None)
p.polar(r, theta, color="navy", alpha=0.5, line_width=2)
output_notebook()
show(p) 22. Donut Chart from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
source = ColumnDataSource(data=dict(labels=labels, values=values))
p = figure(title="Donut Chart Python Bokeh Example by PratikPathak.com", x_range=labels)
p.wedge(x=0, y=1, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_field="labels")
p.wedge(x=0, y=1, inner_radius=0.3, outer_radius=0.4, start_angle=0.6, end_angle=2.6, color="white")
p.axis.visible = False
p.grid.grid_line_color = None
output_notebook()
show(p) 23. Treemap from bokeh.plotting import figure, show, output_notebook
from bokeh.models import TreemapRenderer, ColumnDataSource
import pandas as pd

# Sample data
data = pd.DataFrame({
    'label': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H'],
    'parent': ['', 'A', 'A', 'B', 'B', 'C', 'C', 'D'],
    'value': [10, 5, 4, 3, 2, 1, 1, 1]
})

source = ColumnDataSource(data)
p = figure(title="Treemap Python Bokeh Example by PratikPathak.com")
treemap = TreemapRenderer(
    source=source,
    label='label',
    parent='parent',
    value='value',
    color_discrete_sequence=["#e6550d", "#3182bd", "#31a354", "#756bb1", "#636363", "#9e9ac8", "#de2d26", "#fd8d3c"],
)
p.add_layout(treemap)
output_notebook()
show(p) If you get any error, make sure you have installed Python pandas by using the “pip install pandas” command 24. Network Plot from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, NodesAndLinkedEdges
import networkx as nx

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
edges = [('A', 'B'), ('A', 'C'), ('B', 'D'), ('B', 'E'), ('C', 'F')]
G = nx.Graph()
G.add_nodes_from(nodes)
G.add_edges_from(edges)

node_positions = nx.spring_layout(G)
node_x = [node_positions[node][0] for node in G.nodes()]
node_y = [node_positions[node][1] for node in G.nodes()]
node_color = ['red', 'green', 'blue', 'yellow', 'purple', 'orange']

source = ColumnDataSource(data=dict(
    x=node_x,
    y=node_y,
    color=node_color,
    node=nodes
))

p = figure(title="Network Plot Python Bokeh Example by PratikPathak.com", x_range=(-1.1, 1.1), y_range=(-1.1, 1.1))
p.add_tools(HoverTool(tooltips=[("Node", "@node")]))
p.circle('x', 'y', size=15, color='color', source=source)

edge_x = []
edge_y = []
for start, end in edges:
    x0, y0 = node_positions[start]
    x1, y1 = node_positions[end]
    edge_x.append(x0)
    edge_x.append(x1)
    edge_x.append(None)
    edge_y.append(y0)
    edge_y.append(y1)
    edge_y.append(None)

p.multi_line(edge_x, edge_y, color="black", alpha=0.5, line_width=1)

output_notebook()
show(p) 25. Sankey Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Sankey, ColumnDataSource

# Sample data
flows = [
    ('A', 'X', 5),
    ('B', 'X', 3),
    ('A', 'Y', 2),
    ('B', 'Y', 4),
    ('X', 'C', 4),
    ('X', 'D', 3),
    ('Y', 'C', 2),
    ('Y', 'D', 4)
]

source = ColumnDataSource(data=dict(
    source=[flow[0] for flow in flows],
    target=[flow[1] for flow in flows],
    value=[flow[2] for flow in flows]
))

p = figure(plot_width=800, plot_height=400, title="Sankey Diagram Python Bokeh Example by PratikPathak.com")
sankey = Sankey(source=source, label_text_color="white", label_text_font_size="8pt")
p.renderers.append(sankey)

output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 26. Chord Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, CategoricalColorMapper
import numpy as np

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
connections = [
    ('A', 'B', 0.5), ('A', 'C', 0.3), ('A', 'D', 0.2),
    ('B', 'C', 0.4), ('B', 'D', 0.3), ('B', 'E', 0.3),
    ('C', 'D', 0.2), ('C', 'E', 0.3), ('C', 'F', 0.5),
    ('D', 'E', 0.2), ('D', 'F', 0.3), ('E', 'F', 0.4)
]

source = ColumnDataSource(data=dict(
    start=np.array([con[0] for con in connections]),
    end=np.array([con[1] for con in connections]),
    value=np.array([con[2] for con in connections])
))

p = figure(title="Chord Diagram Python Bokeh Example by PratikPathak.com", plot_width=800, plot_height=800)
chord = p.chord(
    source=source, 
    start_angle=np.pi / 2,
    direction="clockwise",
    colors=CategoricalColorMapper(factors=nodes, palette="Spectral11")
)

output_notebook()
show(p) How to Contribute? Feel free to open a PR request on our GitHub repo. Steps to contribute: Fork the repo Make changes in the Forked repo and save Open a Pull Request That’s it 😄! 🌿 Contribute Conclusion In this Repo, I have shared 25+ Python Bokeh examples that can help you learn Python Bokeh. Feel free to contribute to our GitHub repo and keep it updated. Also published . here

Getting Started With Python Bokeh: 25+ Data Visualization Examples With Source Code

The Dark Side of Open Source: Securing the Software Supply Chain

This paper is available on arxiv under CC 4.0 license. Authors: (1) Evan Ning, Milton Academy & evan_ning24@milton.edu; (2) David R. Kaeli, Department of Electrical and Computer Engineering & Northeastern University kaeli@ece.neu.edu. Table of Links Introduction Sieve Optimizations Mathematical Optimization Implementation Results and Discussions Conclusions, Source Code & References 6 Conclusions In this paper, we explored how to better parallelize the execution of the Sieve of Eratosthenes. Clearly, both the arithmetic complexity of computation and the storage requirement play a big role in the computation speed of the standard implementation. However, our proposed algorithm is faster and more compact (requiring N/3 auxiliary storage), resulting in a more efficient parallel execution of the Sieve of Eratosthenes. We explored selecting the chunk size to optimize memory usage, which can avoid memory thrashing. Our incremental segmented approach enables us to handle huge numbers, almost approaching infinity. This is crucial for large-scale prime number computations and applications that require extensive use of primes. Memory management in high-performance multi-threaded computations is extremely important. The cache sizes and physical memory constraints can have a huge impact on performance. We encountered this when selecting the size of our scratch memory. What we found is that there are tradeoffs in terms of selecting this size. Next Steps for advancing the Multithreaded Sieve Algorithm Implementation: Prime Data Store File Writing: The current setup is poised for indefinite expansion. However, to efficiently manage and store results, there is a need to implement writing to files, ensuring the preservation of computed data. To address potential memory constraints, the prime base list will be segmented. Each segment will contain up to 125 million primes to prevent memory overflow and ensure efficient retrieval. Custom Big Integer Library: In order to sieve infinitely, a special library for large numbers needs to be created. While there are default big integer libraries, we can customize our own for further optimization. For example, we only divide by 6, so a special operation can be created. Furthermore, a special operation can be created to efficiently mod 6. Incorporation of Multiprocessing: To further optimize the computational process, the implementation of multiprocessing will be explored. This approach is to divide the tasks into multiple computation nodes. However, this comes with its set of challenges. One primary concern is minimizing data transfer between processes, ensuring coherence and consistency in results. Source Code GitHub: https://github.com/evanning1/Multi-thread_Prime_Sieve References [1] R.L. Rivest, A. Shamir, L. Adleman, “A Method for Obtaining Digital Signatures and Public-Key Cryptosystems,” (1978) Massachusetts Institute of Technology, https://people.csail.mit.edu/vinodv/COURSES/MAT302-S12/Rsapaper.pdf [2] Xuedong Luo, “A Practical Sieve Algorithm Finding Prime Numbers,” (1989) Communications of the ACM, 32(3), March 1989, 344–346. https://doi.org/10.1145/62065.62072 [3] Torben Hansen, “Parallel Implementation of the Sieve of Eratosthenes,” F120116 Utrecht University - Institute of Mathematics, https://himsen.github.io/pdf/Project1_parallel_algorithms_Torben.pdf [4] Mário Cordeiro, “Parallelization of the Sieve of Eratosthenes, Faculdade de Engenharia da Universidade, s/n 4200-465 Porto PORTUGAL . [5] Jonathan Sorenson, “An Introduction to Prime Number Sieves,” Computer Sciences Technical Report #909, Department of Computer Sciences University of Wisconsin-Madison, January 2, 1990. [6] POSIX Threads Programming https://hpc-tutorials.llnl.gov/posix/ [7] Research Computing at Northeastern University https://rc-docs.northeastern.edu/en/latest/

Memory Efficient Multithreaded Algorithm: Conclusions, Source Code & References

Open Source Software (OSS) has revolutionized the way software is developed, distributed, and utilized, offering a for innovation and advancement. community-focused platform While from the surface, the field of OSS, with its focus on collaboration and social diversity, may appear as a level playing field, where contributors from all over the globe collaborate on projects driven by a shared passion for software technology and innovation, beneath this lies a challenging issue– gender biases in OSS persist throughout the overall dynamics of community engagement, negatively influencing the experiences of women, their participation rates, collaboration & contribution, and leadership opportunities. Exploring Gender Dynamics in the OSS Industry The underrepresentation of women in Open Source Software communities is a , with various studies highlighting the stark gender disparities in participation rates. not-so-secret and well-documented phenomenon Although there’ve been numerous efforts from communities and organizations to promote diversity, inclusion, and more female participation, these efforts have time and time again been proven futile, with women remaining a minority of open source developers . at just 11% This becomes even more alarming when we consider the fact that women are better , highlighting how facing numerous barriers hinders their advancement within these communities. represented in newcomers to OSS Studies have established that women face unique and substantial challenges when starting their OSS journeys or contributing to particular OSS projects. Some of these are gender-specific limitations that impede their involvement and integration in open-source communities. Specifically, research has found that women often experience . These obstacles significantly contribute to the lack of female representation in the field as they make it unwelcoming for them and thus hamper their growth within such spaces. communication problems, non-friendly and less accommodating orientation procedures, and prejudices based on their personal attributes while participating in OSS projects Women’s contributions are often because of these systemized biases. Such prejudice in evaluative procedures not only undermines women’s contributions but also perpetuates an exclusionary culture within OSS communities that negatively affects women’s interest in participating and contributing to open-source projects. scrutinized more than men’s and cast with greater suspicion Getting involved in an open source is tough enough for first-timers, but when confronted with additional challenges such as having their contributions undervalued or dismissed as well as finding it more difficult to get recognition compared to their male peers, women may be less interested in participating in OSS activities. Another reason why women are less likely to contribute to open-source software is the existence of gender biases in OSS as a never-ending cycle that excludes them perpetually and . Women who aspire to be contributors may feel discouraged or hesitant due to a lack of role models, mentors, leaders, or individuals they can relate with. acts like a curse that keeps on cursing The absence of representation and community leads to isolation and alienation which prevent women from seeing themselves as part of the OSS community thus hindering the recruitment and retention of women within OSS projects while reinforcing ideas that OSS is a male-dominated field where women are not welcome. Open Source Collaboration Can’t Work Without Demographic Diversity The systematic prejudices and gender inequalities that exist in Open Source Software communities have a negative impact on the ecosystem as a whole, affecting participation and collaboration in these settings. It is not difficult to see how a in the development process can result from the underrepresentation of women and gender prejudices in OSS. lack of various viewpoints and experiences Diversity is essential for promoting innovation and creativity in virtually every industry because people with varied experiences offer unique perspectives and methods for approaching problem-solving. Women are underrepresented in open source software (OSS) projects thereby losing out on important contributions that could improve the caliber and efficacy of the software being developed. Gender biases not only misalign with the principles upon which open-source communities are built but have also been shown to result in underperformance and reduced effectiveness. Various studies indicate that more diverse teams bring about greater innovation and success for technology companies. According to a , businesses with executive teams in the top quartile for gender diversity were shown to be 15% more likely to achieve above-average profitability compared to those in the bottom quartile. 2017 report Furthermore, done by Boston Consulting Group and the Technical University of Munich showed an increase in leadership team diversity led to an overall increase in innovation, as well as better financial performance in both emerging markets and developed economies – again affirming that growing numbers of diversified management help industries innovate and firms boost their profit margins. another research The far-reaching implications of gender disparities and biases in OSS communities make it so that an industry already plagued with doesn’t come close to solving its problems. By embracing demographic diversity, including gender diversity, OSS communities can tap into a wealth of unique perspectives, experiences, and problem-solving approaches, leading to more innovative and effective open-source projects. the difficulty of finding skilled professionals with relevant expertise OSS communities must prioritize diversity and inclusion efforts to foster environments where all individuals, regardless of gender, feel welcomed, valued, and empowered to contribute to their fullest potential. Only by making strides to fix the lack of demographic diversity can open-source software and technology at large truly fulfill its promise of collective progress and innovation. Putting the 'Open' in Open Source Software: Addressing Bias and Fostering Inclusivity Taking a deliberate and proactive approach toward fostering gender inclusivity in the OSS space involves creating a supportive community, developing platforms, infrastructure, and tools that help mitigate biases and challenges faced by women in OSS communities, and integrating said tools. Inclusive Guidelines: Creating inclusive guidelines means setting out that set out the expected conduct and values within the OSS society. These guidelines foster OSS’s openness by explicitly prohibiting discrimination, harassment & biased behavior and emphasizing the importance of respect, inclusivity & diversity Only by setting clear expectations and consequences for violating these guidelines, OSS communities can create a safer and more welcoming environment for all members. clear policies and procedures Education, Collaboration, and Networking: Encouraging collaboration, networking opportunities, and educational training & resources for women in the OSS community is crucial for breaking the cycle of underrepresentation and exclusion while also fostering a more inclusive environment. Virtual meetups, conferences, forums, and community events not only provide platforms for women new to OSS to connect, share knowledge, and collaborate on open-source projects but also serve as avenues for accessing educational training and resources. OSS communities can introduce women leaders and role models by organizing such interactions and availing training resources so that young females will be able to see them instead of assuming that they do not exist. Such an all-rounded approach will ultimately result in more diverse OSS communities capable of supporting women with essential resources for their success in the OSS ecosystem. Bias Mitigation Algorithms and Methods: Implementing algorithms and mechanisms within OSS platforms, such as GenderMag, to mitigate biases in evaluation processes can significantly contribute to breaking the cycle of under-representation of women and fostering a more inclusive environment by nipping the bud of detected biases early on. GenderMag is a method that helps incorporating transparent and fair evaluation criteria. By utilizing such tools, OSS platforms can ensure that contributions are evaluated based on merit rather than gender, thus reducing the impact of unconscious biases. Additionally, by incorporating bias mitigation algorithms directly into OSS platforms, such as through automated tools for detecting and addressing implicit biases, OSS communities can establish fairer evaluation processes. Transparent and fair evaluation criteria help ensure that contributions are assessed objectively, regardless of the contributor's gender. development by identify and address gender biases in software Continuous Improvement and Feedback: Regularly reviewing and improving community processes, including contribution evaluation and decision-making procedures, is paramount to addressing gender bias. By consistently refining these processes and seeking input from diverse voices, OSS communities can create more equitable and welcoming environments that encourage the active participation of women. Creating technology that can support women on OSS platforms necessitates an ongoing commitment to improvement and the need to gather feedback, particularly from female users. This entails regularly seeking contributions, having user tests carried out, and refining designs and features based on client responses; thus ensuring that the OSS tools and platforms adequately serve the interests of women. By incorporating these strategies, both technical and community-focused, into the OSS communities and the design and development of OSS platforms and tools, technology can be built to accommodate women and create a more inclusive and welcoming environment for all individuals in OSS communities. Conclusion among participants in these communities, there are still specific challenges that have made it difficult for most women to join or progress. To address this problem, preventive measures must be instituted. Despite several initiatives to promote diversity This includes establishing regulatory frameworks for broader participation, fostering cooperation amongst members, creating supportive social networking functions within online platforms that can help identify gender biases in online environments, and providing educational materials together with other resources aimed at addressing such issues while continually refining procedures that promote collective progress. Feature Image Source: Pexels

Bytes and Bias– Unraveling the Influence of Gender Dynamics in Open Source Software

Although AI, in this article mostly referring to LLMs, seems to be revolutionizing many areas and many laws of conventional technologies that don’t seem to apply to it, I am surprised how many concepts of traditional cybersecurity are appearing in a new light in the era of AI. One that scares me the most is the concept of “Sources and Sinks.” What Are Sources and Sinks? The concept of sources and sinks originally came from security code reviews. It is in reference to the fact that data comes from somewhere, a.k.a. the source, (user input, databases, website components, other systems, etc…) and flows through the application and the logic that processes it into a so-called sink (database, webpage element, an email, etc…). Security researchers commonly do something called “Taint Tracking” or “Taint Analysis” to identify what data goes where. This is already pretty hard to do in large-scale applications, but with enough effort, it is achievable. But at least those applications follow deterministic algorithms, which means, that if A happens, the consequence is always B. You may already see where this is going. AI “Sinks” This Concept You get it? Because of sources and sinks and how it is hard to do taint analysis in AI. Well, the problem with AI is twofold. One thing is that LLMs are intentionally non-deterministic in the conventional sense. Because they are imitating human language and thinking to some extent, they have built-in measures, like temperature and sampling methods, that add randomness to the output of an LLM. Which means, especially in cases where an LLM makes a decision about what tool to use or where to put some data, you can not be entirely sure what exactly an LLM will do. Of course, when you only have one agent, that can either use a calculator or browser, this is not a huge problem. It is, however, unlikely that this is what AI systems of the future will look like. Although, if you add email and PowerPoint to this agent, it can probably do 90% of a McKinsey consultant’s work ;). The other thing is that AI systems will probably have a lot of sources and sinks because one of the intended use cases for these systems is to replace human workers, and if you are an office job person, you know what an insane amount of information you need access to and how much stuff you are allowed to do (at least from a typing stuff into something perspective). Now this becomes even scarier when you realize that, even though this concept is known to more and more developers, it is insanely often ignored. This may be because of a lot of microservice stuff and compartmentalized responsibility, where the individual developer just takes some data that some other developer sends them, and who knows where the user puts in what and where it ends up, or simply because of a lack of awareness. But now imagine these same developers building large-scale, “non-deterministic” applications with access to literally the entire internet and confidential corporate data, and they can use a bunch of tools, send emails and Slack, surf the web and someday control heavy machinery. And as if this was not enough, they have to do all this under immense time pressure with technology that not even the people who created it fully understand and libraries that abstract it to a level where nobody knows what is going on under the hood. Is All Hope Is Lost? I “hope” that this is not the case, and I think now is the perfect time to put a spotlight back on this “old” concept of “Sources and Sinks.” Currently, everyone is out there building some PoC implementation of some AI tooling, and when everyone sees the potential of this crazy new invention, it is very likely that the PoC just gets some more bells and whistles and a feature here and a new API there, and before you know it, it manages some crucial process in your company. So please, while it is still easy, and you think you know where all your data is coming from because it is just this little tiny PoC, start modeling your data flows. Because if you don’t do it now, somebody will have to reverse-engineer them later. It won’t be you because you have all this other stuff going on. It will be some dude in a basement, and he will do some involuntary pen-testing for you. Talking ‘bout AI. This has probably already crossed your mind: “Can’t LLMs just do the data flow modeling and taint tracking?” I think this is very likely the case. This will be an amazing field for AI security companies to work on; you will probably be able to hack a small-scale solution together in a couple of hours. I might even do it one of these days. But just because AI COULD do it, it does not mean that it is not important to give this some thought. Because hackers won’t wait till the market is full of tools that help secure AI. They are in fact hacking AI right now, like crazy. What do you sink about it?

AI Security — What Are Sources and Sinks?

Remember when code was just lines on a screen, without personality or dialogue? Well, buckle up, fellow developers and AI enthusiasts, because the landscape is changing at breakneck speed! Large Language Models (LLMs) are bursting onto the scene, harnessing the power of natural language processing to transform the way we code, create, and communicate. Yan Lekun, the director of Meta AI, says, "The potential of LLMs is huge; they can help us automate tasks, create content, and translate languages," but because there are so many open options, choosing the right LLMs is blindfolded, It’s like walking through a jungle gym. Don't worry, brave developers! This is where I come in, your friendly local AI sherpa, ready to guide you through the hottest open LLMs that will supercharge your career and unleash your inner AI wizard. But there are so many options. Choosing the right LLM can feel like walking into a jungle gym with your eyes closed. Fear not, brave developers! I'm here to act as your Sherpa, guiding you through the hottest open-source LLMs to increase your productivity and unleash your inner AI wizard. So, let's crack open the treasure chest and unveil the top 10 gems: No introduction needed. Bard, Google’s behemoth LLM, boasts an impressive 137B standard, making it a powerhouse for handling complex NLP tasks. While still in development, Bard showed early promise in areas such as rule generation and summaries. Bard (Google AI): Coming in hot at 178B parameters, Jurassic-1 Jumbo is a monster LLM with a reputation for amazing realistic speech understanding. If your project requires you to work with large datasets of text and code, Jurassic-1 Jumbo could be your new best friend. Jurassic-1 Jumbo (AI21 Labs): Together, NVIDIA and researchers at Megatron developed a monster 530B-parameter LLM specifically geared to generate a variety of creative texts, such as poetry, code, writing, pieces of music, emails, documents, and more etc. If you want to add a bit of AI-powered creativity to your project, this is the LLM for you. Megatron-Turing NLG (NVIDIA): This multilingual marvel, backed by a consortium of AI researchers, boasts impressive performance in 46 languages. Need to translate code comments, document international projects, or build multilingual chatbots? BLOOM is your one-stop shop. BLOOM: Clocking in at a whopping 1.75T parameters, WuDao 2.0 is a Chinese superpower LLM that focuses on multilingual capabilities. This model is particularly adept at understanding and responding to natural language in Chinese. WuDao 2.0 (BAAI): This community-driven project offers a smorgasbord of open-source LLMs of various sizes, all trained on a huge dataset of text and code. EleutherAI is a good choice for developers of different LLMs who want to experiment with construction design to find the one that best suits their specific needs. EleutherAI (EleutherAI): Another gem from EleutherAI GPT-J is a 6B parameter LLM based on the OpenAI GPT-3 framework. Although smaller than some of the other options on this list, GPT-J offers a good balance of power and functionality, making it a great choice for developers with limited resources. GPT-J (EleutherAI): Although not strictly open-source, Cohere provides a free layer to their powerful LLM APIs. This makes it a good choice for developers who want to try out LLM before diving headfirst into their models. Cohere (Cohere Corp): LaMDA, short for Language Modeling for Conversational Programming, is Google’s take on the conversational AI model. LaMDA is specifically designed to create realistic and engaging interactions, making it ideal for developers of chatbots or virtual assistants. LaMDA (Google AI): Rounding out our list is Cicero, a real language LLM from Meta AI. Trained in the extensive use of information and law, Cicero excels at such tasks as interviewing and taking information. Cicero (Meta AI): Conclusion: Your Personal AI Playground Awaits Note that this list is just a taste of the vibrant and ever-growing world of open LLMs. Each model has unique strengths and weaknesses, so experiment, explore, and find the one that fits your specific needs and projects. As Fei-Fei Li, executive director of the Stanford Human-Centered AI Institute, aptly puts it, "AI Think of these LLMs as powerful tools in your developer toolbox, in order to maximize your creativity.” has made your work easier, what." Expect to push the limits of possibility and open your eyes. So, dive in, explore, and prepare to be amazed at the potential of an open-source LLM to transform your growth journey. The future is bright and powered by AI, but the story of how it unfolds is yours to write, code by code, with the help of these wonderful language models.

10 Open-Source LLMs That Will Rock Your Dev World in 2024

How do you choose the most appropriate model for your application? An analysis of evaluating and building applications on open-source large language models. The computational complexity of AI models is growing exponentially, while the compute capability provided by hardware is growing linearly. Therefore, there is a growing gap between those two numbers, which can be seen as a supply and demand problem. On the demand side, we have everyone wanting to train or deploy an AI model. On the supply side, we have Nvidia and a number of competitors. Currently, the supply side is seeing earnings skyrocket, and the demand side is stockpiling and vying for access to compute. It’s a simple formulation but one that works well to explain the current situation in AI. If anything, the generative AI explosion in the last couple of years has validated this formulation. How can this supply vs. demand imbalance be resolved then? One way would be to introduce , and some people are working on that. If you are not one of them, the other option would be to work on the software side of things. That is, to try and come up with ways to make AI model training and inference less computationally intensive. more AI hardware with more capabilities This is what CEO and co-founder Yonatan Geifman has been working on since 2019. Deci is focused on building AI models that are more efficient and accurate. Initially, , later adding a range of language models to its portfolio. Deci Deci released computer vision models Interestingly, most of these models are open source. We caught up with Geifman to explore the landscape of open-source AI, aiming to address some key questions. Do the benefits of open source and the criteria to consider in the classical build vs. buy dilemma apply to AI large language models (LLMs)? And how can LLMs be evaluated and tailored to the specific needs organizations have for their applications? https://youtu.be/1GD9J84Bw34?embedable=true Open source Large Language Models are different To begin with, let’s examine what “open source” means in the context of LLMs. Open-source software has been around for a long time and is sufficiently well understood. , and different licensing schemes have their peculiarities. Open source has gone through some turbulence Still, in software, things are relatively straightforward. It’s all about the source code and its availability. Having the source code and the right dependencies and environment means being able to build and deploy software. But what about AI models? Yes, we also have the source code. However, there are additional artifacts on the way from source code to deployment. We have the dataset used to train the model. We have the training process and . Some model releases also include metadata such as metrics and . Can the same open-source definition apply? weights for the source code model cards Geifman believes that the most important artifacts are the source code and the weights, as having those shared enables people to use and build on top of other people’s work and collaborate around those derivatives that are open source. Metadata is nice to have but not essential in his view. But there’s also the crucial issue of data and training process. Deci, like Mistral, is not sharing data or details about the process of training their open-source models. Geifman noted that there are trade secrets around those. To get a model that works well, it’s not just about having the right data but also about using the right parts of the data in the right way, he added. Geifman thinks that sharing the data and the training process is significantly less important than sharing the model itself. By contrast, that “trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point.” OpenAI’s James Betker claims What probably everyone will agree to is the fact that open-source software and open-source models are not exactly the same thing. As well as the fact that if open sourcing of model architectures and software had not happened, AI would not be at the point where it is today. Either way, there is a whole spectrum from having everything open to having everything closed, and most models fall somewhere in between. Geifman noted that there is a tendency in the community to use the term models instead of open-source models. open weights Open source Large Language Models vs. GPT-4 and Claude Open weights or open source, however, what’s probably more important is how these models can be used by organizations. This is where things look more like traditional software. Organizations have many choices. They can use open-source models hosted in their own infrastructure and in some third-party providers' infrastructure, or they can use hosted, closed-source models such as GPT-4 or Claude 3 via an API. What Geifman sees is that many organizations embark on their AI journey by leveraging closed-source model APIs. He thinks the reason is it’s easier to get started this way, as there is less investment in time and expertise. However, over time, that may not be the more sensible approach. Prices may fluctuate, but as closed-source APIs come with a premium, costs will start adding up. But this may not even be the most serious drawback. Closed source models are black boxes controlled by a third party, and there are implications to this. When things go well, using a thing party model may mean lowering the bar to entry and having peace of mind. But the flip side is that if the model is compromised, goes down, or suffers from a bug or degradation in performance, applications that use it will suffer the consequences. Similar effects will also occur even when updating models, as their behavior may change inconsistently. Prompt engineering is brittle enough as it is, so adding this parameter makes building applications on top of closed-source LLMs an exercise in non-determinism. There are also privacy and security concerns to be aware of, as sensitive data may be exposed. This is the reason why some organizations are not only refraining from using closed-source models but outright banning their employees from using them, too. Large Language Model customization: Fine-tuning vs. RAG Most of the above is not specific to AI models or LLMs. What is specific, and more often than not required for real-world applications, is customization. Whether it’s via , LLMs need to be tailored to users' needs and datasets. fine-tuning or RAG Even though fine-tuning and RAG aim to achieve similar goals, the way they work is quite different, as Geifman pointed out. Fine-tuning changes the actual model for a specific data set, while RAG focuses on injecting data into the generation phase. Geifman finds that with RAG, there is not a lot of difference if you’re looking to apply it to an open-source or a closed-source model. Fine-tuning, on the other hand, will be much easier to do for an open-source model. In the debate of fine-tuning versus RAG, Geifman thinks the future is a combination of both. “There are some benefits in using RAG, mostly on data privacy. You don’t want to train the model on private data, but in RAG you can supplement the model with private data in the prompt. That’s one of the benefits of using RAG, and also the fact that you don’t need to fine tune the models all the time in order to have updated data”, Geifman said. It’s worth mentioning that both fine-tuning and RAG are constantly evolving, with techniques such as and , respectively. Overall, however, Geifman thinks that building more complex use cases will be much easier to do in open source versus closed source in the future, especially when the accuracy gap is bridged. LoRA+ Corrective RAG The reference to accuracy brings us to another multi-variant equation, namely, how to evaluate LLM performance. At the moment, as Geifman mentioned, it seems that closed-source models such as OpenAI’s GPT-4 or Anthropic’s recently released Claude 3 have an edge in terms of accuracy over open-source models. But there is a lot of nuance here. Evaluating Large Language Models Geifman predicted that the gap will be closed by the end of 2024. However, as has recently been demonstrated, carefully fine-tuned models specializing in specific tasks can outperform larger models for these tasks. LoRA Land, a collection of 25 fine-tuned open-source Mistral-7B models, was shown to consistently outperform base models by 70% and GPT-4 by 4-15%, depending on the task. What’s more important, however, is evaluating performance holistically. Typically, people tend to equate LLM performance with accuracy. While accuracy is an important metric, there are many other parameters that need to be considered as well. These all revolve around the nature of the application for which the LLM will be used. Even a typical use case such as document summarization, for example, has different requirements depending on the type of document and what the summary is used for. Latency and throughput are also very important, again depending on the application requirements. Some applications may be more tolerant than others. And then there’s , obviously. Even though this is not a metric per se, it’s something that needs to be factored in as well. cost Geifman’s recommendations? First, if you need speed, you’d better work with smaller, fine-tuned, tailored models for your applications. Second, for most B2B applications, you don’t need GPT-4-level accuracy. If you’re trying to summarize documents, you can do it with . mid-sized language models with good performance, lower latency, and better value for money As to how to evaluate models holistically, that’s still an unsolved problem. The two main alternatives at the moment are human evaluation and using an LLM to evaluate another LLM. Both have pros and cons and as usual, there are lots of nuances and blind spots. You might be tempted to note that it’s all fine and well, but not everyone needs to be familiar with all the nuances of LLM evaluation. There is always the , which has become the de facto evaluation ground for all LLMs. Just like any other benchmark, however, these metrics are only indicative. Hugging Face leaderboard The bottom line is that if you want to assess whether a model is cut out for the task at hand, you can stand on the shoulders of giants, but you’ll have to put in the work yourself. And that means formulating your requirements, developing realistic evaluation scenarios, and putting LLMs to the test. Market outlook In the end, the fact that best practices and principles from other areas also apply at least partially to LLMs is probably good news for organizations looking to incorporate LLMs in their applications. As Geifman noted, the excitement from LLMs and their capabilities and the potential of integrating them in enterprises across many verticals has energized the market. Geifman shared that Deci’s AI platform has seen its market more than triple in 2023. Deci’s market is divided into two segments, each with its own characteristics. One of them is computer vision, which is more mature. People know what they need, and they are looking towards production use cases with multiple use cases per company and team, as Geifman put it. Then, there is the GenAI segment, in which people are more in an exploratory phase. Geifman believes 2024 will be the year we’ll see many GenAI-based projects going into production. He sees document summarization and customer care as the dominant use cases. https://pod.co/orchestrate-all-the-things-podcast-connecting-the-dots-with-george-anadiotis/evaluating-and-building-applications-on-open-source-large-language-models-featuring-deci-ceo-co-founder-yonatan-geifman?embedable=true Deci addresses both computer vision and GenAI with open-source models. includes SuperGradients, a library for training PyTorch-based computer vision models, DeciLM 6B and DeciLM 7B, which Geifman noted offers similar accuracy to Mistral 7B with better efficiency and performance, the DeciCoder 1B and DeciCoder 6B code generation models as well as the DeciDiffusion text-to-image model. Deci’s offering These models are free to use, with additional tools to customize, fine-tune, and deploy them. Deci’s deployment environment for LLMs is called Infery-LLM. It’s an open-core approach in which the models are open source, and the value-add tools are commercial. To showcase the value that Infery-LLM brings, Geifman shared that running DeciLM 7B with and without Infery-LLM gets 4,5x vs. 2x the performance of Mistral 7B, respectively. Deci is also working on releasing more LLMs with more parameters and better performance. Also published . here

SOURCE

source.network

-- employees

Since n.d.

COMPANY RANKING

SOURCE

EVERGREEN INDEX #9911

Representing this company? Click here to claim and customize this page!

Source WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

SOURCE

source.network

-- employees

Since n.d.

COMPANY RANKING

SOURCE

EVERGREEN INDEX #9911

RELATED COMPANIESRANK

STORIES ABOUT SOURCE

Representing this company? Click here to claim and customize this page!

Source WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!