Scribble Data is an advanced analytics data product company

Scribble Data

The world of data has changed dramatically over the past decade. Traditional databases, which were designed to store information in a structured format, have evolved into massive warehouses of unstructured data that sit on multiple servers across different locations. Not too long ago, we were used to seeing monolithic systems dominated by behemoths, the likes of Oracle and IBM. If you are an analyst or business user who needs access to this type of data—and who doesn't?—it meant slow-moving systems that were incredibly difficult to manage. The birth of a new software stack The increasing complexity of systems eventually drove the need for modern software stacks that could help organizations run complex applications while managing to stay cost-effective. The open source movement helped with this, by dramatically decreasing the cost of putting complex applications together such as Elastic Search for full-text search and PyTorch for modeling. Robust packaging and operations of the software improved usability, stability, and economics of the system. The (MDS), which has seen a lot of traction over the past decade, builds on the open source movement and is a collection of ideas, tools and methodologies intended to build the enterprise data stack.​ Modern Data Stack Challenges in scaling MDS In the 2010s, we saw rapid adoption of open source tools within the MDS. However, post their initial success a lot of organizations’ initiatives around these ran into challenges when it came to scaling them: The due to the number of tools, configurations, methodologies, and interactions that organizations and teams had to keep up with was overwhelming, leading to burnout and high attrition rates among talent cognitive overload The associated with these technologies was incredibly steep. One has to understand that most of these open source tools were built at sophisticated organizations such as Netflix, Google, and Uber and don’t necessarily suit the needs of organizations that have smaller deployments–a fraction of the scale. learning curve The pace of innovation in the space also meant . With the pace at which newer, better, faster, more efficient tools were arriving on the scene engineers had to learn and unlearn rapidly. shorter lifespans for newer technologies The data science community is one that has several conflicting viewpoints, resulting in a (what’s best for their business). More often than not, the only way to overcome this challenge is by building, which is not only expensive but time-consuming. lack of clarity around what approach one needs to adopt If you’ve been following hype cycles such as Gartner, it’s probably no surprise for you to learn that (which arrives much faster than it did probably a decade ago). Technologies like Hadoop, NoSQL, and Deep Learning which were considered “hot” not too long ago have already passed their peak of the Gartner hype cycle. technology investments have an end-date Points #1 and #2 have played a major role in adding to the stress levels in the industry, and also limiting the talent available for adopting and using technologies. We’ve seen a similar trend in the DevOps space, with the supply of developer talent not meeting the demand for new digital services. about this problem - which has been leading to high burnout, and the average career span of a professional developer being less than 20 years. He recently posted a thread where he did a deep-dive into the complexity in the developer-led landscape, and we can’t help but notice several parallels between what he claims and the MLOps space.​ Tyler Jewell of Dell Capital has been quite vocal Points #3 and #4 highlight the plight of today's data folks–if solving problems wasn’t enough, they end up spending more time trying to figure out “how” they can proceed and solve problems without being able to give much thought to what needs to be done, or the expected outcome. A change is coming… We’re seeing a shift in the data tools used by organizations, driven by an increased recognition that many of them have no choice but to rely on third-party vendors for their infrastructure needs. This is not only due to budget constraints but other constraints as well, such as data security and provenance. In addition, there’s an increased demand for automated processes that allow enterprises to easily migrate workloads from one provider to another without disrupting operations or causing downtime. We’re seeing the effects of these within industries like financial services where data management is often critical for success (for example, credit rating agencies). As a result of all these as well as the challenges listed above, there have been several developments in the community: ​ Organizations are increasingly emphasizing the need to build , giving rise to tools that focus on data quality and data governance. trust in their data There is growing emphasis on Machine Learning and Data Science initiatives being tied to outcomes, and business models that are explicitly . aligned to specific business use cases Ever increasing cost and complexity are resulting in consolidation through feature extensions, acquisitions and integrations. Snowflake, for example, is rapidly growing its list of partners to become a full analytical application stack. Considering the complexity post model deployment, we’re seeing an emergence of tools such as , which help estimate model performance, detect drift, and improve models in production through iterative deployments. We’re seeing this as a way for businesses to close the loop between the business, data, and the model. NannyML A new organization, , emerged to bring together the essential building blocks for Artificial Intelligence applications. They’ve been working on building a , which aims to drown the noise created due to the introduction of a plethora of tools that claim to be the “latest and greatest”, and help non-tech companies level up fast. AI Infrastructure Alliance Canonical Stack for Machine Learning The definition of the MDS is being extended to include data products, apps, and other elements. This is making MDS full-stack. and services are emerging that slice the space based on target users (e.g. data scientists vs. analysts), skill availability, and time to outcome realization. New products The MDS user base is expanding to include the analytics teams and business users. This is resulting in improved user experience, low-code interfaces, and automation. And finally, we’re seeing an emergence of approaches such as the “ ”, which is essentially a deconstruction of the MDS and the MLOps stacks. These approaches emphasize relevance for the business as well as the downstream consumption of generated features to produce business value. Post-Modern Stack ​ What this means Consolidation of tools and platforms, simpler platform developments, and the use of managed services is happening across the industry. This is stemming from the need for businesses to cope with complexity. It’s an exciting time to be a part of this space, and I can’t wait to see how the landscape evolves over the course of the year. At Scribble Data, (the company that I have co-founded) we are keenly aware of this evolution as it is happening. We focus on one specific problem - feature engineering for advanced analytics and data science use cases. This problem space has steadily grown in terms of importance and has evolved in ways that are consistent with the above points. With the right technology mix and solution focus, it is possible to align product value to use cases, while achieving 5x faster time to value (TTV) for each use case. Also Published Here

Growing Data Infrastructure Complexities: Cost Implications and the Way Forward

Crecientes complejidades de la infraestructura de datos: implicaciones de costos y el camino a seguir

Complexités croissantes de l'infrastructure de données : implications en matière de coûts et voie à suivre

डेटा इन्फ्रास्ट्रक्चर की बढ़ती जटिलताएं: लागत प्रभाव और आगे का रास्ता

増大するデータ インフラストラクチャの複雑さ: コストへの影響と今後の展望

Complexidades crescentes de infraestrutura de dados: implicações de custo e o caminho a seguir

Sự phức tạp ngày càng tăng của cơ sở hạ tầng dữ liệu: Hàm ý về chi phí và con đường phía trước

日益增长的数据基础设施复杂性：成本影响和前进之路

Back in 1991, former US Air Force pilot and noted strategist John Boyd after Operation Desert Storm. He noted that three basic elements were necessary to win wars - “People, ideas, machines — in that order!” called for U.S. Military reforms That order remains true even today, and not just in the context of military thinking but in business decision-making as well. Boyd knew that an existing technology alone cannot be responsible for the success of new technology; rather, it needs the alignment plus intersection of technology and the people, their ideas, as well as the organizational / business context. Our fascination with technology often leads us to solutions that are very good in theory like the F-111 or F-35, but not quite as effective on the battlefield, at least not like the F-16, which is still considered one of the most versatile fighter aircraft of all time. The rules aren’t any different for data science The past decade has seen data becoming the competitive differentiator for almost every industry, be it retail, real estate, finance, education, healthcare, entertainment, manufacturing, and more. This is, in part, due to data and technology becoming more accessible. As a result, companies have started investing in data infrastructure to maximize ROI on data science spend, build flexible data operations, and accelerate innovation. So much so, that on Machine Learning and Big Data tools, or what the industry calls the , which is an assembly of these cloud-native tools that reflect how data should flow through an organization for it to be useful, at scale and by multiple users. they spend over $200 billion Modern Data Stack However, according to Gartner, despite all these investments in maximizing ROI on ML spends, . only 53% of projects make it from prototypes to production Despite all the investment – money, time, resources, and valuable data science bandwidth– why is this number so low? Over the past few months, I’ve spoken to several practitioners and end consumers of data science (I’m refraining from calling this a survey because of the small sample size and the anecdotal approach to these conversations), and came back with several observations: Not only is data science today not actively involved in the business of moving the business outcomes, but some data scientists don't even see it as their job. As a result, the models, by the time they’re production-ready, are irrelevant to the business. What’s ironic is that the modern data stack, whose goal is to help organizations save time, effort, and money through better utilization and integration of data tools, is creating bottlenecks in the consumption of data. Data science is getting siloed. Many data science teams drive up solution complexity with their desire to emulate Big Tech. They see themselves as mini-Ubers, and reusing methods that are not fit for context. As a result, they end up spending a lot of time on building technology, which comes a little too late. Data science teams are building technology instead of solving problems. that allow it to be fit for context across the diversity of enterprises and objectives - small and large, high and low skill, accuracy vs cost, and other standard business considerations. Data science is missing ideas/strategies Understanding and fixing these is critical to the data science function’s impact. Business is Data Science’s Business Before there was  big data, it was generally accepted that different departments would create and manage their own data, and use it in meaningful ways to achieve their goals. However, as the complexity of data increased, we started seeing new big data tools emerging which needed specialized skills. Eventually, this meant data science becoming increasingly disconnected from the business functions or their day-to-day use case requirements. Beyond the tools, the cultural disconnect between data science teams and business has only grown. Not only are the skills of the staff involved different, but their approaches are remarkably different. While the business teams are focused on the day-to-day operations and identifying ways to improve processes to improve efficiency, profitability, and more, the data science teams typically have a more R&D flavor in their approach. We witnessed conflict between the business and data science teams around purpose, process, and outcomes in a number of organizations. This is not just a waste of time and resources but is also detrimental to the future of these organizations. Data science is a core business function, and teams need to be accountable for driving organizational and business outcomes. Once tied to the results, the success of this function will ensure a much higher success rate for all data science initiatives. Data Science Teams’ Choices Are Driving Up Complexity Shiny object syndrome or "SOS'' is a core part of human nature, where we tend to focus on what’s the latest and greatest - the latest tools and technology, being ahead of the curve in the adoption of the modern data stack. All the while disregarding how useful it may actually be. Unfortunately, the hierarchy proposed by John Boyd has been flipped on its head by most organizations, with technology taking precedence over ideas, and finally people. As a result, they tend to adopt new tools without thinking about the consequences, or how they would fit into their context. This becomes the single biggest roadblock. All the time goes towards experimenting with and maintaining complex technologies. There is little time spent in understanding impact, or exploring alternative approaches. There is a Gap in Ideas According to noted data scientist, , there are still a lot of  'ML at reasonable scale' or companies. He notes, “These companies stand to get their first wins with ML and generally have pretty low-hanging fruit to get those wins. They don’t even necessarily require these super-advanced sub-millisecond latency hyper-real-time pieces of infrastructure to start leveling up their machine learning.” Mihail Eric Sub-ML If I was to go back to the military analogy, think about the US Air Force and how they still use aircrafts such as F-16s – aircraft that aren’t the latest and greatest, but serve the purpose better for most requirements. One of the key reasons being that these were aircraft that were designed keeping the needs of the user (pilot) first. Building an ML model isn’t enough, it needs to deliver in the context of the end-user. Similarly, the decision regarding the tooling also needs to consider the requirements of the business and the end-users. (Fun fact: John Boyd’s ideas had an impact on the design and acquisition of other major weapons systems including the A10 Warthog and Bradley Fighting Vehicle, and also on Operation Desert Storm) Strategy First, Technology Later Every organization today is turning into a data company, but with data increasing in volume, depth, and accessibility across organizations — it has become the key to sustainable competitive advantage. However, more often than not, big data is not used well. Organizations are better at collecting data – product usage, customer data, competitor data, etc than designing the strategy around its usage. This brings us to another key element that Boyd spoke about - ideas. Data science being as dynamic as it is, requires ideas that benefit various stakeholders and functions within the organization. This will not only eliminate the siloed nature of data and decision sciences today but also help teams understand the nature and complexity of the problems, as well as the effort required to solve them. This is the key to scaling, and making sure everyone in the organization is aligned on the value that data science brings. I’ve already started seeing some of these examples at play - for example, Scribble Data’s (the company that I co-founded) feature store was used at a Fortune 100 company by their People Analytics team to improve the way a number of key decisions were taken. Data scientists, business managers, and HR specialists worked together to improve the employee experience, incentive alignment, and churn modeling to help retain the best talent within the company. These impacted multiple areas such as hiring, performance management, and employee engagement, all of which fall in the top right corner of the urgent/important matrix, given the Great Resignation. Data Science, by its very nature, is evolving and uncertain project - almost like a university project where all you have is either a fuzzy idea of the goal, or a clearly defined goal that doesn’t offer much value to the business. It’s important to think of clear, realistic ideas along with key stakeholders to define the priorities for the data science team in order to see more success in projects, and buy-in from the leadership team. The ML Stack Should be Fit for Purpose Every year, billions of dollars are spent on data science tools that are overengineered and don’t serve the purpose of 90% of the organizations or their use cases. As a result, the incremental cost of complexity is much more than the incremental value of these tools. Here are a few points to consider when you’re building your data stack: Fit business constraints: Make your tooling investment decisions based on their expected outcome, which can be measured on a regular basis Right complexity: Identify tools that are fit for purpose, which will not only make the lives of the people using them easier but will also make data deeply relevant to individuals in the value chain Take things slow: Avoid making decisions based on what’s new and trending, or would look good on paper. For example, an engineer I recently interviewed discussed how his former organization spent an inordinate amount of resources building real-time infrastructure whereas it provided only the smallest uptake, and was applicable to only 1% of their use cases. Most importantly, the goal of the data stack is to eliminate data silos within the organization, which is why they all need to fit together. Better integration between these tools will make the data stack more verticalized and fit for purpose, offering significantly more value to organizations and stakeholders.

Data Science Teams are Doing it Wrong: Putting Technology Ahead of People

Turn your bad drawings into detailed works of art Have you ever had a hard time translating your scribbled drawings into detailed and clear images? Fear not, is here to help! This AI model can take your rough sketches and turn them into fully realized images with a high level of detail. In this guide, we’ll explore the ins and outs of ControlNet Scribble, learn how to use it, and even find other similar models using Replicate Codex. ControlNet Scribble About the ControlNet Scribble Model ControlNet Scribble is an image-to-image created by the developer . As mentioned earlier, it can take your rough sketches and turn them into highly detailed images. This model uses a technique called deep learning, which involves training the model on large amounts of data to allow it to learn patterns and relationships within the data. ControlNet Scribble is based on a neural network architecture that has been optimized to produce high-quality outputs. AI model jagilley According to the Replicate Codex website, ControlNet Scribble is ranked at 5 out of all the models on the platform. This means that it is a popular and widely used model that has been highly rated by its users. Additionally, its creator jagilley is ranked 3 out of all the creators on the platform, making them a highly respected and experienced developer in the AI space. Understanding the Inputs and Outputs of ControlNet Scribble Inputs ControlNet Scribble requires the following inputs: An image file: This is the sketch or scribbled drawing that you want to turn into a detailed image. Prompt string: This is a short phrase or sentence that provides additional information or context about the image you want to generate. Num_samples: This parameter specifies the number of samples to generate. Higher values may cause out-of-memory errors. Image_resolution: This parameter specifies the image resolution to be generated. Ddim_steps: This parameter sets the number of steps for the optimization process. Scale: This parameter sets the guidance scale. Seed: This parameter sets the seed for the random number generator. Eta: This parameter sets the eta for DDIM. A_prompt: This parameter specifies the added prompt. N_prompt: This parameter specifies the negative prompt. Outputs The output of ControlNet Scribble is an array of image URLs that represent the detailed images generated from the input sketches. A Step-by-Step Guide to Using ControlNet Scribble If you want to use ControlNet Scribble, you can interact directly with the model’s demo on Replicate via their UI. Here’s a step-by-step guide to using ControlNet Scribble via the Replicate API: Step 1: Install the Replicate Node.js Client First, you need to install the Replicate Node.js client. You can do this using the following command: npm install replicate Step 2: Authenticate with Your API Token Next, you need to authenticate with your token. You can do this by setting your API token as an environment variable: API export REPLICATE_API_TOKEN=[token] Step 3: Run the Model Finally, you can run the model using the following code: import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});const output = await replicate.run(
  "jagilley/controlnet-scribble:435061a1b5a4c1e26740464bf786efdfa9cb3a3ac488595a2de23e143fdb0117",
  {
    input: {
      image: "path/to/your/image/file",
      prompt: "a short phrase or sentence that describes your image",
      num_samples: "1",
      image_resolution: "512",
      ddim_steps: 20,
      scale: 9,
      seed: 0,
      eta: 0,
      a_prompt: "best quality, extremely detailed",
      n_prompt: "longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality"
    }
  }
); You can specify the path to your input image file, as well as the prompt and other parameters, in the input object. Once you run the model, the output will be an array of image URLs that you can use to view the generated images. If you’re not comfortable with coding, you can still use ControlNet Scribble via the demo link provided earlier. The UI is user-friendly and allows you to input the necessary parameters and generate images without any coding. Taking it Further — Finding Other Image-to-Image Models with Replicate Codex Replicate Codex is a great resource for discovering AI models that cater to various creative needs, including image generation, image-to-image conversion, and much more. It’s a fully searchable, filterable, tagged database of all the models on Replicate, and also allows you to compare models and sort by price or explore by the creator. It’s free, and it also has a digest email that will alert you when new models come out so you can try them. If you’re interested in finding similar models to ControlNet Scribble, here’s how you can do it: Step 1: Visit Replicate Codex Head over to Replicate Codex to begin your search for similar models. Step 2: Use the Search Bar Use the search bar at the top of the page to search for models with specific keywords, such as “image-to-image” or “drawing.” This will show you a list of models related to your search query. Step 3: Filter the Results On the left side of the search results page, you’ll find several filters that can help you narrow down the list of models. You can filter and sort models by type (Image-to-Image, Text-to-Image, etc.), cost, popularity, or even specific creators. By applying these filters, you can find the models that best suit your specific needs and preferences. For example, if you’re looking for an image restoration model that’s the cheapest, you can just search and then sort by cost. Conclusion In this guide, we’ve explored the ControlNet Scribble model, learned how to use it, and even discovered other similar models using Replicate Codex. I hope this guide has inspired you to explore the creative possibilities of AI and bring your imagination to life. Thanks for reading! Subscribe for more! Also published here.

Converting Sketches to High-Quality Images with ControlNet Scribble AI

The Essential Architectures For Every Data Scientist and Big Data Engineer 

… for most companies. Over the years, I’ve heard a lot of people talk about algorithmic business and how big data will improve business strategy. For most, I strongly suspect it won’t. I originally wrote this post in and what I’ve observed since then makes me feel it is worth repeating. To explain the problem, I’m going to have say certain things that many will find uncomfortable. February 2015 Chess and business I want you to imagine you live in a world where everyone plays Chess against everyone else. How good you are at Chess really matters, it determines your status and your wealth. Everyone plays everyone through a large computer grid. Ranks of Chess players are created and those that win are celebrated. Competition is rife. The oddest part of this however is that no-one in this world has actually seen a Chessboard. When you play Chess in this world, you do so through a and it looks like this … control panel Each player takes it in turn to press a piece. Each player is aware of what piece the other player pressed. White starts first. Then Black counters, then White, then Black and so the game continues building up the sequence of presses. The game continues until at some point, usually after a long sequence of presses, the system announces that someone has won or it is a draw. Unbeknownst to either player, there is actually a board and when they press a piece then a piece of that type is randomly selected. That piece is then moved on the board to a randomly selected position from all the legal moves that exist. White’s opening move might be to select ‘Pawn’, one of White’s eight pawns would then be moved one or two spaces forward. But all of this is hidden from the players, they don’t know any of this exists, they have no concept there is a board … all they see is the control panel and the sequence of presses. However, we have the sequence! Within that sequence people will find ‘favourable’ patterns e.g. the best players seem to press the Queen as often as possible! People will come up with their own favourite combinations and try to learn the combinations of other winners! “When your opponent moves the Knight, you should counter with Queen, Queen, Rook” etc. People in this world would write books on secret knowledge such as or . As you aggregate more and more data from different games in ever larger “big data” systems and AI is applied then the more patterns will be proclaimed and the more secrets of success announced in popular publications. You can already imagine that article in Harvard Business Review collecting sequences from over 100,000 games and announcing the killer combination of . “The art of the Bishop” “The ten most popular sequences of Successful Winners” Queen, Queen, Pawn, Knight Remember this isn’t because people are daft, they simply have no concept of the board in this world. Some through experience might even gain a vague notion that there is a landscape, a board on which we are playing. Let us suppose one of those people — by luck, by accident or by whatever means — twigs that a landscape does exist and somehow finds a way to interact with it. Imagine one day, you play that individual but they can see something quite remarkable … the board. You start off planning to use your favourite opening combination of You make your first move. Pawn, Pawn, Knight. They counter with , you make your next move, they counter and you’ve lost. Pawn you scream. Obviously, this was just luck! But just in case this is a new secret sequence, you scribble down their presses. “ , “What the fiddlesticks happened there!” Pawn Queen — wow what a simple sequence!” But every time you play this player, you lose and you lose fast. You keep recording their sequences and even try to use their sequence against them. They beat you in two presses — Pawn (b) and Queen (b) (known as Fool’s Mate) or they beat you in four presses — Pawn (b), Queen(b), Bishop(b), Queen(b) (known as Scholar’s mate) but you can’t replicate their success even though you copy their sequences, it never works and you keep on losing. People will be amazed by the incredible skill of this player but also confused that they can’t replicate the success. They will start to seek all different forms of answers to explain what is happening. Maybe it’s not just the sequence but the way that the player presses the button? Maybe it’s timing? Maybe it’s what they had for lunch? Maybe it’s their attitude, a cultural thing? They seem to be happy (because they’re winning). Is happiness the secret to winning? All sorts of correlations and speculations will be jumped upon. However there is no luck to this or special way of pressing buttons. The player controlling Black simply has far greater situational awareness. The problem for the player controlling White is they have no real understanding of the context of the game they are playing. White’s is just a shadow of the landscape and the sequence of presses lacks any positional information or consistent understanding of movement on the board. When faced with a player who does understand the environment then no amount of large scale data analysis on combinations of sequences of presses through the or application of artificial intelligence or algorithms that is going to help you. control panel control panel I need to emphasise this point of understanding the landscape and the importance of situational awareness and so we’ll now turn our attention to another story — Themistocles. Themistocles SWOT The battle of Thermopylae (the tale of the three hundred) and the clash between the Greeks and the mighty army of Xerxes has echoed throughout history as a lesson in the force multiplier effect of landscape. Themistocles devised a strategy whereby the Athenian navy would block the straits of Artemisium forcing the Persians along the coastal road into the narrow pass of Thermopylae where a smaller force could be used to hold back the vastly greater Persian numbers. I’ve provided a map of this below. Now, Themistocles had options. He could defend around Athens or defend around Thebes but he chose to block the straits and exploit the landscape. Each of Themistocles’ options represents a different on the map. In the same way that our enlightened Chess player has many to move pieces on the board. By understanding the landscape you can make a choice based upon one move is better than another. This is a key point to understand. WHERE WHEREs WHY People often talk about the importance of in strategy but is a relative statement i.e. here over there. To answer the question of you need to first understand and that requires situational awareness. WHY WHY WHY WHY WHERE Now imagine you’re in that Greek army. Themistocles had turned up on the eve of battle, he had enthused the troops with purpose and a goal — “ . Let us assume the Greek army (a collection of state armies) had varying but overall decent principles e.g. working as a team, being skilled in what you do, collaborating with each other and were well trained. But now imagine, that no-one on the Greek side had a concept of maps or even an understanding of maps. We need to defend the Greek states against the horde of Xerxes” Themistocles steps forward and tells the troops to have as he announces his great vision in the form of a SWOT (strengths, weaknesses, opportunities and threats) diagram. “no fear, we are the greatest warriors in the world” How confident would you feel? I know I’d be running for the hills. Now in terms of combat, I’d hope you’d agree that a strategy based upon an understanding of the landscape is going to be vastly superior than a strategy that is derived from a SWOT. Equally from the point of view of communication and learning then a map beats a SWOT hands down. The question you need to ask yourself however is — what do we most commonly use in business? SWOT or MAP? On business and situational awareness The problem that exists in most businesses is that they have little or no situational awareness. Most executives are even unaware that you can map the business environment and increase situational awareness. They are usually like the Chess Players using the Control Panel and oblivious to the existence of the board, relying on magic secrets of success from other players — “I hear Amazon just did this, we should do the same” I say oblivious because these people aren’t daft, they just don’t know that a landscape exists. But how can we be sure this is happening? Language and behaviour are often our biggest clues. If you take our two Chess players (White with little or no situational awareness, Black with high levels) then you often see a marked difference in behaviour and language. Those with high levels of situational awareness often talk about positioning and movement. Their strategy is derived from WHERE and so they can clearly articulate WHY they are making a specific choice. They tend to use visual representation (the board, the map) to articulate encounters both present and past. They learn from those visualisations. They learn to anticipate others moves and prepare counters. Those around them can clearly articulate the strategy from the map. Those with situational awareness talk about the importance of WHERE — , . They describe how strategy is derived from understanding the context, the environment and exploiting it to your favour. The strategy always adapt to that context. “we need to drive them into the pass” “we need to own this part of the market” Those with low levels of situational awareness often talk about the HOW, WHAT and WHEN of action (e.g. the combination of presses). They tend to focus on execution as the key. They poorly articulate the WHY having little or no understanding of potential WHEREs. They have little or no concept of positioning and they often look to ‘copy’ the moves of others — . Any strategy they have is often gut feel, story telling, alchemy, copying, magic numbers and bereft of any means of visualising the playing field. Those around them often exhibit confusion when asked to describe the strategy with any precision beyond vague platitudes. Those who lack situational awareness often talk about the importance of WHY? They describe how strategy should be derived from your vision (i.e. general aspirations). They often look at you agog when you ask them “Company X gained these benefits from implementing a digital first strategy, we should do the same” “Can you show me a map of your competitive environment?” Back in 2012, I interviewed 160 different high tech companies in Silicon Valley looking at their level of strategic play (specifically their situational awareness) versus their use of open (whether source, hardware, process, APIs or data) as a way of manipulating their environment. This was part of a study for the and we considered that these companies were the most competitive of the competitive. Leading Edge Forum What the study showed was there was significant difference in the levels of strategic play and situational awareness between companies. When you add in a tendency towards action and performance measure by market cap changes over the last seven years then those with high levels of strategic play had performed vastly better than those with low levels. Leading Edge Forum 2012 study on situational awareness versus action Now this was Silicon Valley and a select few. I’ve conducted various interviews since then and I’ve come to conclusion that less than 1% of companies have any mechanism of visualising their environment and continuously learning from it. The vast majority suffer from little to no situational awareness but then they are competing in a world against others who also lack situational awareness. This is probably why the majority of strategy documents contain a tyranny of action and why most companies seem to duplicate strategy memes and rely on backward causality. We should be digital first, cloud first, we need a strategy for big data, social media, cloud, insight from data, IoT … yada yada. It’s also undoubtably why companies get disrupted by predictable changes and are unable to overcome their inertia to the change. At the same time, I’ve seen a select group of companies and parts of different Governments use mapping to remarkable effect and discovered others who have equivalent mental models and use this to devastate opponents. Now, I don’t believe we’re anywhere near the pinnacle of situational awareness in business — far better methods, techniques and models will be discovered than those that exist today. Which brings me back to the title. In the game of Chess above, yes you can use large scale data analytics, AI and algorithms to discover new patterns in the sequences of presses and certainly this will help you against equally blind competitors. Such techniques will also help you in business improve your supply chain or understand user behaviour or marketing or loyalty programmes or operational performance or any number of areas in which we have some understanding of the environment. But this won’t help you in strategy against the player with better situational awareness. Most business strategy itself operates in a near vacuum of situational awareness. For the vast majority then I’ve yet to see any real evidence to suggest that big data is going to improve this. There are a few and rare exceptions but in general, the key is first to understand the landscape and that a landscape exists. Alas for most, it will become hunting for that magic sequence … Pawn, Pawn … what? I’ve lost again? If you’re going — — it’s fairly easy. To help you on your way, , all creative commons on just how to do that with a technique that I’ve been using for over a decade. Please feel free to help yourself. “Well, how do you map a business?” I’m writing a book Attribution-ShareAlike 4.0 International Originally published at blog.gardeviance.org on January 8, 2017.

Why big data and algorithms won’t improve business strategy

Judgement Day.

Security continues to be one of the most important aspects of web3. As a developer, you should be securing your smart contracts by ensuring that testing—specifically a wide variety of testing—is integrated into your smart contract development lifecycle. If you aren’t embedding testing in your flow, you’re just biding your time until your smart contracts are hacked. Let’s look at how fuzzing meets that need, and how you can quickly implement fuzzing by using the Diligence Fuzzing for your Foundry web3 projects. newly released support Web3 security standards are still emerging Web3 security standards are still being defined. We need more tools, more best practices, and more (I hate to say it) processes. Not only do we need to catch up to traditional development practices, we need to surpass them. After all, a in our smart contracts. lot of money is at stake How do we contribute to growing and maturing security in web3? By building better tools, integrating more processes into our daily practice, and automating as much as we can. We have a long way to go to catch up with the automated CI/CD pipelines of traditional development … but we’re getting closer. One step that’s a good start: integrating and automating fuzzing into your smart contract development cycle with Foundry and Diligence Fuzzing. Let’s look at what that means and how to actually do it. What is Diligence Fuzzing (Fuzzing-as-a-Service)? Diligence Fuzzing (aka Fuzzing-as-a-Service) is a web3 fuzzing tool for testing smart contracts by ConsenSys. Using a spec of the expected behavior of your code, it creates a large number of inputs/transactions that might violate this expected behavior. It then runs the fuzzed inputs against your code, trying to cover as much code as possible, and trying to, well, make things fail! If your code violates the specification, the fuzzer raises a warning. The specs are created using Scribble to annotate your smart contracts. According to the Diligence site, Diligence Fuzzing is, “The most powerful fuzzer to find bugs and vulnerabilities before they are exploited by bad actors on mainnet.” What is Foundry? Foundry is an , used for creating and deploying smart contracts on Ethereum/EVM-compatible blockchains. It has lots of components: a CLI, a testing framework, and a development framework. open source development framework From : “Foundry manages your dependencies, compiles your project, runs tests, deploys, and lets you interact with the chain from the command-line and via Solidity scripts.” their own book Foundry has recently become very popular. I love it because it was built with security and testing in mind. Interestingly, Foundry ships with a fuzzing tool that works pretty well. Foundry does a lot of things pretty well—that’s another reason I love it! But Diligence Fuzzing was built , and it takes your fuzzing game to the next level. A few quick reasons: just for fuzzing It uses gray-box testing as opposed to black-box testing, essentially allowing the fuzzer to discover and test more code/paths. More code coverage, of course, allows you to find more bugs in your code. In a recent comparison, than Foundry. it found 25% more violations By default, it uses “stateful fuzzing” where it generates of transactions instead of just one transaction. This can hit edge cases in the logic where multiple transactions are needed to trigger the bug. sequences It’s faster! And now existing Foundry projects can use Diligence Fuzzing with just a few steps. How Fuzzing enhances your smart contract development cycle Fuzzing is a type of software testing considered “property-based testing.” This type of testing checks that the output of your code matches expectations for a wide variety of inputs. A property-based tester automatically creates a whole lot of random inputs and runs them through your code, trying to find one that violates the expected output. This kind of testing is usually a complement to your unit testing. Your unit tests check for known flows, simple inputs, and edge cases you’ve already caught and fixed … basically, the test cases you know you need to run. Fuzzing, on the other hand, is a software testing technique that looks for defects in your source code that you don’t know about. It automatically runs through a lot of possibilities (more than you could ever test for manually), trying to find complex and subtle bugs that you wouldn't otherwise uncover. Fuzzing in action Let’s walk through the steps needed to start fuzzing a Foundry project with Diligence Fuzzing. First, install the by running the following on your command line (also be sure you’re on Python 3.6 or newer): Fuzzing CLI pip3 install diligence-fuzzing You need a and an API key. , then . Diligence fuzzing account Sign up here go to your dashboard to create the key Now run: echo FUZZ_API_KEY='your api key here' > .env This will add your API key to a .env file for the CLI. Finally, run the fuzzing CLI and create a new fuzzing campaign. fuzz forge test That’s it! You can target specific tests and contracts using the following options: Usage: fuzz forge test [OPTIONS]

  Command to:  * Compile unit tests  * Automatically collect unit-test contracts  * Submit to fuzzing

Options:

  -k, --key TEXT         API key, can be created on the FaaS Dashboard.

  --dry-run              Outputs the data to be sent to the FaaS API without making the request.

  --match-contract TEXT  Only run tests in contracts matching the specified regex pattern

  --match-path TEXT      Only run tests in source files matching the specified glob pattern

  --build-args TEXT      Additional string of `forge compile` command arguments for custom build strategies (e.g. --build-args=--deny-warnings --build-args --use 0.8.1)

  --help                 Show this message and exit. For more information, . check out the documentation here Please note a couple things: Everything above assumes you have installed Foundry and written some fuzzing tests. Since your existing Foundry tests probably contain several Solidity assertions and invariants, the fuzzing tool can just use those to start breaking things. Only some of the Foundry cheat codes are currently supported: ‘warp’, ‘prank’, ‘deal’, ‘roll’, . Additional support is planned soon. and a few more Keep security top of mind with Fuzzing and Foundry Securing web3 smart contracts is critical, and it has been for a long time. But with the advent of new testing tools, new integrations, and more rigorous processes, we’re getting closer to a reliable, secure, and mature ecosystem. Fuzzing is one step towards that goal. Also published . here

Scribble Data

scribbledata.io

-- employees

Since n.d.

COMPANY RANKING

SCRIBBLE DATA

EVERGREEN INDEX #9344

Scribble Data WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Scribble Data

scribbledata.io

-- employees

Since n.d.

COMPANY RANKING

SCRIBBLE DATA

EVERGREEN INDEX #9344

RELATED COMPANIESRANK

STORIES ABOUT SCRIBBLE DATA

Scribble Data WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!