Are you struggling with Cloud Providers (i

Opsly

opsly

What are Internal Developer Platforms (IDPs)? Internal developer platforms (IDPs) are platforms that developers can leverage to build and deploy their applications to one of the environments. Their main purpose is to increase the pace of development and reduce the developer’s dependence on DevOps and Platform engineers through automation and developer self-service. The platform may be built in-house, usually, an adaptation of an open-source or purchased offering such as platform as a Service (PaaS). The adoption has been both recent and rapid. Puppet’s state of platform engineering finds that over 51% of companies that have adopted Internal Developer Platforms have done it in the past three years. And an overwhelming majority of the respondents (93%) declared that IDP adoption is a step in the right direction. survey Internal developer platforms are good at helping the team manage applications and infrastructure from one place, providing tight integration with the existing tools and services, and giving developers self-serve and collaboration capabilities. Benefits of IDPs IDPs offer several benefits to the organizations using them. The first relates to improvements in infrastructure and IT management, such as increased productivity stemming from reduced communication times between devs and infra teams. Secondly, they reduce the complexity of the cloud and make it easy for people in your org to pick up on the best practices set forward by your team. In larger organizations, they can also be an easy way of managing RBAC to deployments, infra creation and management, and more. Responses from Puppet’s State of Platform Engineering 2023 report also show how IDP has improved : DevOps KPIs An increase in development velocity (68%) 42% say development velocity has improved “a great deal” since they started doing platform engineering Improved productivity (59%) Improved system reliability (60%) Improved security (55%) Components of IDPs Let’s dive deeper into the components of IDPs and discuss which parts of the application and infrastructure management process they improve. IDPs are built by integrating with the existing set of tools and services that the company currently uses. These can be your source control systems (e.g., GitHub, GitLab), Continuous Integration and Continuous Deployment (CI/CD) pipelines, and observability tools such as monitoring and logging systems. Integration with existing tools and services In addition, they may also include some form of access control, observability dashboards, and a set of best practices for developers to use. Application and infrastructure management IDPs can be used to automate a lot of the hassle of deployment. By setting up IDPs with GitOps best practices, users can automatically deploy application and infra changes with each commit to the Git code. Managing multiple environments is another popular use case of IDPs. They can be effective in creating, providing selective access, and managing the use of multiple different environments such as development, testing, pre-prod, and prod. Several tools set up preview environments on the fly. Their ability to monitor and view deployments across environments helps organizations scale dynamically to changing workloads and maintain consistent performance across their infrastructure. IDPs empower developers with self-serve capabilities by offering a centralized, user-friendly interface to access tools, resources, and services on demand. These capabilities facilitate faster development cycles, reduce operations teams’ dependency, and enable greater autonomy in seamlessly creating, testing, and deploying applications. Developer self-service capabilities The streamlined workflow also makes it considerably easier for new employees to get started with your stack without diving into the internal workings. Some tools also add a collaboration layer on top that allows different members of a team to work together easily and review changes before deploying the code. IDPs can achieve enhanced security and compliance by providing Role-based access controls (RBAC), ensuring each team member has the appropriate permissions to access specific resources and perform certain actions. Overall, this minimized the risk of unauthorized access or accidental changes while also promoting collaboration and efficient workflows among cross-functional teams. Collaboration and governance features IDPs help developer teams maintain transparency, accountability, and traceability throughout the development process by providing audit logs and history tracking. This enables easier identification and resolution of issues, as well as ensures compliance with regulatory and organizational requirements for data handling and change management. Platform engineering Companies that decide to build their own platform usually hire platform engineers, who are specialized professionals responsible for the creation and continuous improvement of the IDP. They work to implement and maintain tools, services, and best practices that streamline development processes, ensuring a smooth and efficient workflow for the organization's developers. Platform engineering is the process of designing, building, and maintaining an IDP that provides a centralized, scalable, and efficient infrastructure for developers within an organization. An IDP simplifies and standardizes application development, deployment, and management, enabling faster and more reliable software delivery. Software development trends that are catalysts for IDP adoption Digital transformation: As businesses worldwide increasingly embrace digital transformation, there is a growing need for efficient and robust software development processes. IDPs play a crucial role in enabling companies to stay agile and quickly adapt to the dynamic digital landscape. Need for faster software delivery: In today's fast-paced business environment, organizations must deliver new features and applications at an unprecedented speed. IDPs provide a standardized, automated, and centralized platform that accelerates development cycles, enabling companies to stay competitive and responsive to market changes. The growing complexity of software architectures: With the rise of microservices, containers, and cloud-native technologies, managing and deploying software has become increasingly complex. IDPs help simplify this complexity by providing a unified platform where developers can build, test, and deploy applications with ease, regardless of the underlying architecture. Demand for DevOps and CI/CD practices: The need for IDPs grows as more organizations adopt DevOps and Continuous Integration/Continuous Deployment (CI/CD) practices. IDPs enable seamless collaboration between development and operations teams, automating many manual tasks and ensuring smooth transitions throughout the software development lifecycle. Scalability and flexibility: IDPs offer a scalable solution that can accommodate the growing needs of organizations, regardless of their size. They provide a flexible platform that can be easily customized and adapted to cater to the unique requirements of different teams and projects. Cross-region collaboration: With businesses operating across multiple geographies, the need for a platform that supports cross-region collaboration is essential. IDPs empower development teams spread across the globe to work together seamlessly, enabling efficient knowledge sharing and fostering a culture of innovation. In summary, the rising popularity of IDPs across companies of all sizes and regions worldwide can be attributed to their ability to streamline software development processes, simplify complex architectures, support DevOps and CI/CD practices, and facilitate cross-region collaboration. As the demand for digitalization and agility continues to grow, IDPs are poised to play a critical role in shaping the future of software development. Commercially available internal developer platforms is a DevOps automation platform that helps engineering manage both the app and infra side of things and ship faster! Built with the GitOps best practices in mind, Argonaut reduces the complexities of creating and maintaining cloud setups. Whether it is Kubernetes app deployment to AWS or GCP, there are several runtimes, environments, regions, integrations, and app types to choose from. Argonaut offers a simple way to develop and operate modern cloud applications on Kubernetes. It can be adopted as either a self-hosted or PaaS option. There’s also a with essential plugins, templates, and applications making it easier to get started. Its features benefit not only the developers but also the platform engineers and CIOs. Mia Platform marketplace is an internal developer platform providing simplicity, automation, reusability, and self-service. It acts as a Platform Orchestrator that lets engineering teams remove bottlenecks by enabling them to build code-based golden paths (executable config files and templates) for developers. It can be used through the CLI or UI. Humanitec provides engineering teams self-serve access to tools and information. It helps developers ensure operational excellence across services with its that can be set up in a secure and compliant way. Opslevel integrations is a Kubernetes application management platform that enables efficient deployment processes. Developers can leverage its standardized application and policy definitions that are platform-agnostic. It also has a GUI-based portal to manage apps after deployment and gain visibility into pipelines for smoother operations. Shipa provides a context-rich software catalog with maturity and quality scorecards. It also supports comprehensive developer self-service actions while providing additional role-based access controls (RBAC). Their provides many key features and makes it a worthy contender. Port free-forever , powered by Crossplane, offers an enterprise-grade control plane solution for multi-cloud and hybrid environments, enabling efficient management of cloud infrastructure. The command line and make it more effective and easier to get started. Upbound Up Upbound marketplace is an all-in-one DevOps platform that streamlines the application deployment process. It provides a comprehensive feature set in a modular manner that allows teams to focus fully on business functionality. DevOpsBox Evaluating Internal Developer Platforms Despite its myriad of benefits, Internal Developer Platforms do not make sense for all teams. They can end up being overkill for certain types of engineering teams and a burden to build and maintain for companies with smaller teams. When they don’t make sense You have existing processes that are efficient. Don’t complicate things too early. Companies that are currently using PaaS or other managed offerings should continue doing so as long as possible. Limited resources and team size. This could also mean your team is small, and most of your team is senior and comfortable writing scripts and managing infrastructure. There is low development complexity. If you have just one app with a simple single-cloud setup. And, if your app is monolithic and doesn’t make use of the microservice architecture, there is little benefit from creating an IDP. Incompatible organizational culture. If an organization’s culture is resistant to change or does not foster collaboration and communication, implementing an IDP might not be successful and could even lead to decreased efficiency and productivity. When they make sense You plan to start using Microservices. This usually also means a growing development team size or the complexity of projects you handle. You have a small team, and not everyone feels comfortable with deployments, scripting, and infrastructure, and you have not yet hired a dedicated DevOps. Dependencies on other colleagues are blocking your developers. The cost of your existing setup (such as PaaS) is too high. Which is inevitable once you start to scale to meet new requirements. You have plans to go multi-cloud, adopt more modern cloud services, and scale geographically. You want to increase standardization and consistency across your teams. An IDP can help reduce errors, improve code quality, and ensure all developers work with the same set of tools and follow the same best practices. Also published here.

An In-Depth Guide to Internal Developer Platforms (IDPs)

Una guía detallada de las plataformas internas para desarrolladores (IDP)

Un guide détaillé des plates-formes de développement internes (IDP)

आंतरिक डेवलपर प्लेटफ़ॉर्म (IDPs) के लिए एक गहन मार्गदर्शिका

内部開発者プラットフォーム (IDP) の詳細ガイド

Um guia detalhado para plataformas internas de desenvolvedores (IDPs)

Hướng dẫn chuyên sâu về Nền tảng dành cho nhà phát triển nội bộ (IDP)

内部开发者平台 (IDP) 深度指南

While Silicon Valley is laying off thousands of workers, companies in the smart buildings industry (SBI) are facing serious workforce shortages in almost every field. Businesses of every size, from systems integrators (SI) to equipment manufacturers, are fighting over engineers and developers. It’s the perfect time for laid-off, bored, or disillusioned workers to give the smart buildings industry a serious look. One might assume: “Well, with such a worker shortage, the SBI must be a crap place to work.” Far from the truth. The SBI actually pays well offers challenging, meaningful work. It also holds some of the largest companies in the world, yet it’s an industry often “hidden” from most people. and That’s because building operation is one of those things that, when it works well, go unnoticed. The air conditioning keeps us cool. The lights turn on. The toilets flush. It’s only when these events don’t happen that we notice…and we notice fast. So, while the SBI doesn’t produce a “sexy” ride-share app or disruptive social media platform, it’s an industry with far-reaching and more profound impacts on our lives by comparison. Far from being a boring vocation, building automation is an exciting, innovative industry that’s seeing significant acceleration due to the growth in AI, analytics, and digital twin technologies. In an attempt to bring SBI out of the shadows, here are some facts and ideas that shed light on one of the tech world’s best-kept secrets. Make a Contribution to Saving the Environment Real Buildings are a big part of our lives. We live, work, and sleep within them (when not camping) about 90% of the time. Houses, malls, schools, hospitals, skyscrapers, and other facilities require enormous amounts of space and energy to operate. Together, the built environment has a major impact on the environment. And the impact is growing larger. Consider these stats: actual Buildings make up nearly worldwide. (1) 40% of all greenhouse gas emissions Of that number, come from building operations alone. (1) 27% Global building floor area is expected to . (1) double by 2060 There are close to commercial buildings in the U.S. alone. (5) 6 million But where there’s a big impact, there’s a big opportunity. By automating building systems like lighting, HVAC, and security, we can operate facilities that use fewer resources and emit less carbon. Several studies estimate we could cut energy consumption within commercial buildings by 30% using smart building systems. (2, 4) Think of the environmental implications. However, only commercial buildings in the U.S. have a building automation system (BAS). (3) 10% of the 6 million Tech adoption and innovation can solve the energy efficiency problems of wasteful buildings, and the potential energy savings are real. There is important work to be done, and the engineers and developers who design future smart buildings are key players in reducing GHG emissions. By choosing a career in the SBI, you’ll be on the ground floor of some of the most environmentally impactful work being done today. Listen to an Engineer Describe What It’s Like Working in the Smart Buildings Industry and What His Company Looks for in a New Hire. https://youtu.be/FWZXjeK0gqY?t=648&embedable=true Help People With Accessibility Needs Building automation is a critical part of giving people with special needs better access to facility services. Automatic doors activated by sensors provide easy access for people who have mobility issues. Elevator systems provide access to occupants who can’t use stairs. Assistive technology like automated voices and Braille labels help people with visual impairments navigate spaces. However, these technologies pale in comparison to what’s possible once future technology matures. Imagine designing a smart building that could track the movements of a visually impaired person, guiding and opening doors throughout an entire multi-storied building. Use Cutting Edge Tech Smart building professionals work with cutting-edge technologies like advanced data analytics, digital twin technology, AI/ML, and VR/AR. These advanced tools, along with greater interoperability, present endless possibilities for improving the safety and lives of building occupants. Imagine designing an automated security system that could detect and respond to an active shooter situation. AI and security cameras could provide real-time analysis of a building’s exterior, using facial recognition or similar software to identify threats. The system would then command building access to lock down the facility, set off alarms, and lower lighting and window covers within exterior-facing rooms. The real-time analysis could track first-responder locations and help guide occupants to safe exits away from intruders. Such a system would likely save lives. The future of automation is exciting. Getting into building automation now will ensure you’ll have a hand in shaping the future of how people live, work, and play. Work With Open Systems The name of the game in building automation is interoperability. Devices and software must integrate and share data seamlessly to optimize a building’s operation. Interoperability becomes difficult or impossible when specific companies limit customers to their specific tech. Similar challenges face the smart building industry. For decades, propriety software from a few manufacturers has limited customer choice for the design and maintenance of building systems. However, things have changed dramatically over the past few years, and the industry is moving aggressively towards more open systems and standards that make data transmission and device integration possible. Enjoy Strong Job Security Job security within the smart buildings industry is strong because of multiple factors, including industry growth and the high demand for workers. One report estimates that the global building automation system market will grow at an annual rate of 10% by 2026, from $75.0 billion in 2021 to $121.5 billion. (6) Both developers and engineers are in high demand in the building automation industry, with some occupations like software developers growing at a projected rate of 25%. (7)  The industry is also bracing for a large staffing turnover. Within the next five years, around 35% of the workforce will retire and create an even larger demand. (8)  The result will be higher wages and greater work flexibility. It’s the perfect time to win leverage over negotiating your salary and benefits. Career Opportunities Abound Building automation offers plenty of career growth opportunities. Engineers and programmers can grow their careers rapidly as they gain experience and expertise in the field. Some examples of career growth opportunities within the building automation industry include: : Responsible for designing, implementing, and maintaining building automation systems. Systems Engineer They work closely with clients, contractors, and other stakeholders to develop solutions that meet their needs and are involved in all stages of the project, from design and implementation to testing and commissioning. : Responsible for developing the software that runs building automation systems. They work closely with systems engineers to design and develop software that meets client needs and ensures the smooth operation of building automation systems. Software Engineer/Developer : Responsible for selling building automation systems and services to clients. They work closely with clients to understand their needs and develop solutions that meet those needs. They’re often involved in the design and implementation of building automation systems. Technical Sales : Responsible for overseeing the design, implementation, and commissioning of building automation systems. They work closely with clients, contractors, and other stakeholders to ensure that projects are completed on time, within budget, and to the client’s satisfaction. Project Management : There are also opportunities for engineers and programmers to research and develop new technologies and solutions for building automation systems. Research and Development Conclusion Even though building automation is a “hidden” industry, its impact on our lives and environment is anything but. As a building automation tech, you have the skills and knowledge to shape the future of the built environment. Even though a career in building automation isn’t the flashy “dream job” of Silicon Valley, it is important work that focuses on the needs of humans—where they work and live. fundamental When we design dwellings to be safe, comfortable, and functional, we are directly impacting lives on a level from which everything else flows. That is real change. Sources WHY THE BUILT ENVIRONMENT? (IS-2022-145) Energy Efficiency and Savings for Commercial Buildings Small-and-Medium-Sized Commercial Building Monitoring and Controls Needs: A Scoping Study Report Delves Into the Impacts of Commercial Building Controls on Energy Savings Commercial buildings have gotten larger in the United States, with implications for energy Building Automation Global Forecast to 2027 Bureau of Labor Statistics: Software Developers “Monday Live!” January 16th, 2023. Melissa Boutwell, ASP Brian Collins Brian is the Marketing Manager for OpSys Solutions, an Auckland-based startup that specializes in developing hardware and software solutions for the automated building industry. LinkedIn Facebook Also published here

The Tech World's Best-Kept Secret: Smart Buildings

THE STARS OF SUMMER

Open sourced by Google Research team, achieved wide popularity amongst NLP enthusiasts for all the right reasons! It is one of the best Natural Language Processing pre-trained models with superior NLP capabilities. It can be used for language classification, question & answering, next word prediction, tokenization, etc. pre-trained models of BERT Our case study Question Answering System in Python using BERT NLP and BERT based Question and Answering system demo , developed in Python + Flask, got hugely popular garnering hundreds of visitors per day. We got a lot of appreciative and lauding emails praising our QnA demo. Along with that, we also got number of people asking about how we created this QnA demo. And till the day, we keep getting requests on how to develop such a QnA system using BERT pre-trained model open-sourced by Google. [1] [2] To start with, the readme file on the official GitHub repository of BERT provides a good amount of information about how to fine-tune the model on SQuAD 2.0 but we could see that developers are still facing issues. So, we decided to publish a step-by-step tutorial to fine-tune the BERT pre-trained model and generate inference of answers from the given paragraph and questions on using TPU. Colab In this tutorial, we are not going to cover how to create web-based interface using Python + Flask. We’ll just cover the fine-tuning and inference on Colab using TPU. You can create your own interface using Flask or Django. Overview In this tutorial we will see how to perform a fine-tuning task on SQuAD using Google Colab, for that we will use BERT GitHub Repository, BERT Repository includes: TensorFlow code for the BERT model architecture. Pre-trained models for both the lowercase and cased version of BERT-Base and BERT-Large. You can also refer or copy to follow the steps. our colab file Steps to perform BERT Fine-tuning on Google Colab 1) Change Runtime to TPU On the main menu, click on and select . Set “ “ as the hardware accelerator. Below screeenshot will help you understand how you can change the runtime to TPU. Runtime Change runtime type TPU After Clicking on “Change runtime type”, Select from the dropdown option as given in the below figure. TPU 2) Clone the BERT github repository , or idirectional mbedding epresentations from ransformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. You can find the academic paper here: . BERT B E R T https://arxiv.org/abs/1810.04805 BERT has two stages: and . Pre-training fine-tuning is fairly expensive (four days on 4 to 16 Cloud TPUs), but is a one-time procedure. BERT has released a number of pre-trained models. Most NLP researchers will never need to pre-train their own model from scratch. Pre-training is inexpensive. One can replicate all the results given in the paper, in at most 1 hour on a single Cloud TPU, or a few hours on a GPU. For example, SQuAD can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of 91.0%. Fine-tuning So our first step is to Clone the BERT github repository, below is the way by which you can clone the repo from github. Now get inside the Bert repo using “ “ command cd !git clone https: //github.com/google-research/bert.git cd bert 3) Download the BERT PRETRAINED MODEL BERT Pretrained Model List : BERT-Large, Uncased (Whole Word Masking): 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Large, Cased (Whole Word Masking): 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Base, Uncased: 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Large, Uncased: 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Base, Cased: 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Large, Cased: 24-layer, 1024-hidden, 16-heads, 340M parameters BERT-Base, Multilingual Cased (New, recommended): 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Base, Multilingual Uncased (Orig, not recommended): 102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters BERT-Base, Chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parameters BERT has released and models, that have uncased and cased version. Uncased means that the text is converted to lowercase before performing Workpiece tokenization, e.g., John Smith becomes john smith, on the other hand, cased means that the true case and accent markers are preserved. BERT-Base BERT-Large When using a cased model, make sure to pass -do_lower=False at the time of training. You can download any model of your choice. We have used the BERT-Large-Uncased Model. !wget https: //storage.googleapis.com/bert_models/2018_10_18/uncased_L-24_H-1024_A-16.zip # Unzip the pretrained model
!unzip uncased_L _H _A zip -24 -1024 -16. 4) Download the SQUAD2.0 Dataset For the Question Answering task, we will be using SQuAD2.0 Dataset. (Stanford Question Answering Dataset) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. SQuAD SQuAD2.0 combines the 100,000+ questions in SQuAD1.1 with over 50,000 new, unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. You can download the dataset from SQUAD site https://rajpurkar.github.io/SQuAD-explorer/ #Download the SQUAD train and dev dataset
!wget https: !wget https: //rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json //rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json 5) Set up your TPU environment Verify that you are connected to a TPU device You will get know your TPU Address which is used at the time of fine-tuning Perform Google Authentication to access your bucket Upload your credentials to TPU to access your GCS bucket Using code below you can do the above mentioned 4 points: datetime json os pprint random string sys tensorflow tf os.environ, TPU_ADDRESS = + os.environ[ ]
print( , TPU_ADDRESS) google.colab auth
auth.authenticate_user() tf.Session(TPU_ADDRESS) session:
  print( )
  pprint.pprint(session.list_devices()) open( , ) f:
    auth_info = json.load(f)
  tf.contrib.cloud.configure_gcs(session, credentials=auth_info) import import import import import import import import as assert 'COLAB_TPU_ADDR' in 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!' 'grpc://' 'COLAB_TPU_ADDR' 'TPU address is => ' from import with as 'TPU devices:' # Upload credentials to TPU. with '/content/adc.json' 'r' as # Now credentials are set for all future sessions on this TPU. 6) Create an output directory : You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud Storage) bucket to run the colab file. Please follow the Google Cloud for how to create a GCP account and GCS bucket. You have $300 free credit to start with any GCP product, learn more about it at . Prerequisite https://cloud.google.com/tpu/docs/setup-gcp-account You can create your GCS bucket from here . http://console.cloud.google.com/storage As we are using the Cloud TPU, we need to store the pre-trained model and the output directory in the Google Cloud Storage. If you are not storing it on Bucket you may face the following error : ERROR:tensorflow: recorded training_loop: From /job:worker/replica: /task: :
Unsuccessful TensorSliceReader : Failed to get matching files on uncased_L-24_H-1024_A-16/bert_model.ckpt: Unimplemented: File system scheme '[local]' not implemented (file: 'uncased_L-24_H-1024_A-16/bert_model.ckpt')
[[node checkpoint_initializer_14 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] Error from 0 0 constructor You will get your fine_tuned model in the Google cloud storage bucket after completion of training. For that, you need to provide your BUCKET name and OUPUT DIRECTORY name. BUCKET = BUCKET, output_dir_name = BUCKET_NAME = .format(BUCKET)
OUTPUT_DIR = .format(BUCKET,output_dir_name)
tf.gfile.MakeDirs(OUTPUT_DIR)
print( .format(OUTPUT_DIR)) 'bertnlpdemo' #@param {type:"string"} assert '*** Must specify an existing GCS bucket name ***' 'bert_output' #@param {type:"string"} 'gs://{}' 'gs://{}/{}' '***** Model output directory: {} *****' 7) Move Pretrained Model to GCS Bucket Need to move Pre-trained Model at GCS (Google Cloud Storage) bucket, as Local File System is not Supported on TPU. If you don’t move your pre-trained model to TPU you may face the error. The command allows you to move data between your local file system and the cloud, move data within the cloud, and move data between cloud storage providers. gsutil mv !gsutil mv /content/bert/uncased_L _H _A $BUCKET_NAME -24 -1024 -16 8) Training Below is the command to run the training. To run the training on TPU you need to make sure about below Hyperparameter, that is tpu must be true and provide the tpu_address that we have found above. -use_tpu=True -tpu_name=YOUR_TPU_ADDRESS !python run_squad.py \
  --vocab_file=$BUCKET_NAME/uncased_L _H _A /vocab.txt \
  --bert_config_file=$BUCKET_NAME/uncased_L _H _A /bert_config.json \
  --init_checkpoint=$BUCKET_NAME/uncased_L _H _A /bert_model.ckpt \
  --do_train= \
  --train_file=train-v2 .json \
  --do_predict= \
  --predict_file=dev-v2 .json \
  --train_batch_size= \
  --learning_rate= \
  --num_train_epochs= \
  --use_tpu= \
  --tpu_name=grpc:// : \
  --max_seq_length= \
  --doc_stride= \
  --version_2_with_negative= \
  --output_dir=$OUTPUT_DIR -24 -1024 -16 -24 -1024 -16 -24 -1024 -16 True .0 True .0 24 3e-5 2.0 True 10.1 .118 .82 8470 384 128 True Create Testing File We are creating input_file.json as a blank JSON file and then add the data in the file in the SQuAD dataset format. is used to create a file - is used to write a file in the colab - touch %%writefile You can pass your own questions and context in the below file. !touch input_file.json %%writefile input_file.json
{ : , : [
        { : , : [
                { : [
                        { : , : , : },
                        { : , : , : },
                        { : , : , : }
                    ], : }
            ]
        }
    ]
} "version" "v2.0" "data" "title" "your_title" "paragraphs" "qas" "question" "Who is current CEO?" "id" "56ddde6b9a695914005b9628" "is_impossible" "" "question" "Who founded google?" "id" "56ddde6b9a695914005b9629" "is_impossible" "" "question" "when did IPO take place?" "id" "56ddde6b9a695914005b962a" "is_impossible" "" "context" "Google was founded in 1998 by Larry Page and Sergey Brin while they were Ph.D. students at Stanford University in California. Together they own about 14 percent of its shares and control 56 percent of the stockholder voting power through supervoting stock. They incorporated Google as a privately held company on September 4, 1998. An initial public offering (IPO) took place on August 19, 2004, and Google moved to its headquarters in Mountain View, California, nicknamed the Googleplex. In August 2015, Google announced plans to reorganize its various interests as a conglomerate called Alphabet Inc. Google is Alphabet's leading subsidiary and will continue to be the umbrella company for Alphabet's Internet interests. Sundar Pichai was appointed CEO of Google, replacing Larry Page who became the CEO of Alphabet." Prediction Below is the command to perform your own custom prediction, that is you can change the input_file.json by providing your paragraph and questions after then execute the below command. !python run_squad.py \
  --vocab_file=$BUCKET_NAME/uncased_L _H _A /vocab.txt \
  --bert_config_file=$BUCKET_NAME/uncased_L _H _A /bert_config.json \
  --init_checkpoint=$OUTPUT_DIR/model.ckpt \
  --do_train= \
  --max_query_length= \
  --do_predict= \
  --predict_file=input_file.json \
  --predict_batch_size= \
  --n_best_size= \
  --max_seq_length= \
  --doc_stride= \
  --output_dir=output/ -24 -1024 -16 -24 -1024 -16 -10859 False 30 True 8 3 384 128 To make it easier for you, we have already created a Colab file which you can copy in your Google Drive and execute the commands. You can access the colab file at: . Question Answering System using BERT + SQuAD on Colab TPU If you have any further questions or doubt then please feel free to post them in comments. We’ll get back to you. Previously published at https://www.pragnakalp.com/nlp-tutorial-setup-question-answering-system-bert-squad-colab-tpu/ References Our case study Question Answering System in Python using BERT NLP BERT based Question and Answering system demo

NLP Tutorial: Creating Question Answering System using BERT + SQuAD on Colab TPU

Reasonable security through architecture As a web developer, I am often asked to build pages to collect basic users’ data. The first goal of every advertising campaign, for example, is to collect a large number of emails or phone numbers from people who could be interested in buying some service / product, in order to call them back, contact them, whatever. For many years, sadly but truly, nobody has really cared about how those data were stored in databases. They haven’t been considered “too sensitive” data, still their privacy should have been granted in some way. It was a grey area nobody has ever paid too much attention to. For many years Earth biggest companies haven’t even cared about encrypting , but this is another, possibly even worse, story. If you are interested in some speculation about password encryption and PHP, here’s . passwords of the users another article So guess those (and smaller ones) less sensitive information such as . how much attention companies have been paying to emails or phone numbers Today, after many leaks from famous and less famous companies, and after that kicked in here in Europe, some companies (most of them, to be honest) are starting to be aware of the risks related to storing a large amount of users’ data without any encryption. GDPR A typical , which I found in many old-school projects, is to collect names, mails and phone numbers and store them, , . bad scenario unencrypted in a database located on the same host of the web server This configuration exposes the project to many threats. Especially, if some attackers gain access to database, they can freely retrieve any data of all the users collected by the application. A would be to collect names, mails and phone numbers, them . better scenario encrypt and store in a database hosted on a server different than the web server The should be stored at least on the web server, . encryption keys never on the database The web server should be running a decent Web Application Firewall to keep it as safe as possible. The prototype that I am sharing with you stores the keys in a configuration file inside the public folder of your web server, to keep things simple and let almost everybody install and test the prototype in a super basic LAMP environment hosting. A safer approach would be to , for example. This goal can be reached with little effort for an average experienced developer, and it’s not part of this article. save the keys in a file outside your public folder if you can spend a little bit more on your project, but is to show how to , collecting data. Other, even safer, options are available this article’s goal provide a reasonable safe environment for your small low budget projects “not too sensitive” Obviously, if you are going to store credit cards, medical information, military intelligence and so on, you shouldn’t rely on this simple architecture. A basic working prototype made with PHP and MySql The main problem, when developing an application that encrypts data, is to check or search for those data, once they have been encrypted. We will follow the solution proposed by ParagonIE team in , trying to keep things as simple as possible. this article I will describe how to build a typical landing page that , but still allows us to with the same email or phone , before saving a new user. encrypts private data check if a user has already been stored Moreover, we will build a very basic admin panel (for testing purposes only), with a feature that allows us to in the database, . search for an email or a phone even if they are encrypted Before starting, I am assuming here that you are not using or any other PHP Framework for your small project, in order to keep this article as generic as possible. Laravel pretty that works just out of the box. You can find some information about it in its and in some interesting article . Laravel has a built-in encrypter/decrypter official documentation here This prototype relies on Sodium crypto libraries, which are included as PHP extensions in currently supported versions of PHP (7.1, 7.2, 7.3). To be sure, check your and look for this part. If you can see it, Sodium library is enabled on your server. phpinfo() You can download a on , and preview the . Do not use this prototype as it is in production environments. Use it freely to build your own encryption system and deepen your encryption knowledge. demo source code of my prototype github final result here Basic setup First of all we are going to build a simple html form that should look like this one below. We set all form fields as compulsory, using Bootstrap built-in validation. This form will post data to a script. And that’s it. save.php In folder you can find a , consisting of a single table named Here’s its structure. /_sql dump of the database “subscribers”. subscribers ( ( ) ,
  first_name ( ) ,
  last_name ( ) ,
  email ( ) ,
  phone ( ) ,
  privacy ( ) , ( ) ,
  ip ( ) ,
  email_hash ( ) ,
  phone_hash ( ) ,
  creation_date datetime ) = =utf8; subscribers PRIMARY ( ), email_hash (email_hash), phone_hash (phone_hash); CREATE TABLE id int 11 NOT NULL varchar 255 DEFAULT NULL varchar 255 DEFAULT NULL varchar 255 DEFAULT NULL varchar 255 DEFAULT NULL char 1 DEFAULT NULL source varchar 255 DEFAULT NULL varchar 255 DEFAULT NULL varchar 255 DEFAULT NULL varchar 255 DEFAULT NULL DEFAULT NULL ENGINE InnoDB DEFAULT CHARSET ALTER TABLE ADD KEY id ADD KEY ADD KEY To save those keys as a text, we need to encode them using function, as below. base64_encode() The table fields named and will store the encrypted data, while and are that we will use later to implement search functions on encrypted data. These two fields will store blind indexes, which are basically a keyed hash (HMAC) of the plain text email and phone number. email phone email_hash phone_hash two indexed fields We can import the file to our MySql/MariaDB database. In root folder we can find a little script called . Its task is to generate two random keys for our project. We only need to run this script once, take note of the keys and get rid of the script from our server. generate_key.php The scripts uses function to create the that we will use to encrypt data, and a used to generate a unique hash (the blind index) for email and phone number, in order to make those data searchable in our application. The lengths passed to random_bytes() as parameters are two constants needed by LibSodium to work properly later on. random_bytes() private key blind index key $private_key = random_bytes(SODIUM_CRYPTO_SECRETBOX_KEYBYTES); $index_key = random_bytes(SODIUM_CRYPTO_PWHASH_SALTBYTES); <?php // Generating your encryption key // Generating your blind index key To save those keys as a text, we need to encode them using base64_encode() function, as below. base64_encode($private_key); base64_encode($index_key); <?php echo ?> <?php echo ?> Once we created our , we can write them down in our configuration file, located in folder. If you downloaded the source code from git, you can edit line 50 and 51 of and rename the file as . Don’t forget to data on line 20–23, too. keys for the project /includes config-sample.php config.php add your database connection $private_key = ; $index_key = ; <?php // generate values using /generate_key.php script "" // INSERT PRIVATE KEY HERE "" // INSERT BLIND INDEX KEY HERE We can now publish the project to our web server and check that everything is working, by trying to submit a new subscription to the system. We should see a new record in the database table, as well as on the backoffice demo page. The backoffice page shows the encrypted value actually stored in the database for email and phone number on mouse over. Moreover, if we try to search for an encrypted email among the subscribers, we can find it quickly: A deeper look at the code To understand how this prototype works, we need to have a deeper look at our Data Access Object, in /includes/ dao.php function gets a string as a parameter, in our case email and phone number, and it creates a blind index using our previously generated blind index key and . This will be used of encrypted values in our database. Keeping encrypted values searchable is an interesting matter, well covered by by , where you will also find more advanced solutions. getBlindIndex() sodium_crypto_pwhash() blind index to perform a search this article ParagonIE Here’s the snippet of the function that deals with the creation of blind indexes in our database, to later store the keyed hashes in and table fields email_hash phone_hash {
    $index_key = base64_decode( ->index_key); bin2hex(
        sodium_crypto_pwhash( ,
            $string,
            $index_key,
            SODIUM_CRYPTO_PWHASH_OPSLIMIT_MODERATE,
            SODIUM_CRYPTO_PWHASH_MEMLIMIT_MODERATE
        )
    );
	} <?php public function getBlindIndex ($string) $this return 32 function writes a new record in subscribers table, , and to enable searches on them. Here’s where the encryption happens. The function retrieves the encryption key (l. 47 in ), creates two separate for encrypting email and phone number (l. 50–51), encrypts email and phone number using (l. 53–54) and encode them for safe saving to database (l. 56–57). Here’s a snippet of the funciton. insert() encrypting email and phone number creating two blind indexes dao.php nonces sodium_crypto_secretbox() { ($first_name. == || $last_name. == || $email. == || $phone. == || $privacy. == ) { ;
     } { $email_hash = ->getBlindIndex($email);
      $phone_hash = ->getBlindIndex($phone); $key = ->key;
      $key = base64_decode($key); $nonce_email = random_bytes(SODIUM_CRYPTO_SECRETBOX_NONCEBYTES);
      $nonce_phone = random_bytes(SODIUM_CRYPTO_SECRETBOX_NONCEBYTES); $email = sodium_crypto_secretbox($email, $nonce_email, $key);
      $phone = sodium_crypto_secretbox($phone, $nonce_phone, $key); $email = base64_encode($nonce_email . $email);
      $phone = base64_encode($nonce_phone . $phone); {
        $query = ->db->prepare( );
        $query->bindParam( , $first_name);
        $query->bindParam( , $last_name);
        $query->bindParam( , $email);
        $query->bindParam( , $phone);
        $query->bindParam( , $privacy);
        $query->bindParam( , $email_hash);
        $query->bindParam( , $phone_hash);
        $query->bindParam( , date( ));
        $query->bindParam( , $source);
        $query->bindParam( , $ip);
        $query->execute();
        $id = ->db->lastInsertId(); $id;
      } (PDOException $e) { ;
      }
     }
  } <?php function insert ($first_name,$last_name,$email,$phone,$privacy,$source,$ip) if '' '' '' '' '' '' '' '' '' '' return false else // create blind index for encrypted searchable fields $this $this // get key for encryption $this // get nonce for encryption // encrypt data // encode data for saving into db try $this 'INSERT INTO subscribers (first_name, last_name, email, phone, privacy, email_hash, phone_hash, creation_date, source, ip) VALUES (:first_name,:last_name,:email,:phone,:privacy,:email_hash,:phone_hash,:creation_date,:source,:ip)' ':first_name' ':last_name' ':email' ':phone' ':privacy' ':email_hash' ':phone_hash' ':creation_date' 'Y-m-d H:i:s' ':source' ':ip' $this return catch //echo "Error: " . $e->getMessage(); //exit(); return false ?> Our data access object in has other methods for listing the records in subscribers and count them, but I will focus on two more methods. dao.php The first one is , the function that we need to encrypted . Here’s a snippet dec() decrypt data {
    $decoded = base64_decode($string);
    $key = base64_decode( ->key);
    $nonce = mb_substr($decoded, , SODIUM_CRYPTO_SECRETBOX_NONCEBYTES, );
    $ciphertext = mb_substr($decoded, SODIUM_CRYPTO_SECRETBOX_NONCEBYTES, , ); {
      $plaintext = sodium_crypto_secretbox_open($ciphertext, $nonce, $key); $plaintext;
    } (Error $ex) { ;
    } ( $ex) { ;
    }
  } <?php public function dec ($string) $this 0 '8bit' null '8bit' try return catch return "Cannot decrypt data." catch Exception return "Cannot decrypt data." ?> function gets an encrypted string as input, retrieve the encryption key, separate the nonce from the encrypted text and then decrypt it. dec() This function can be used while we loop our records from table, to decrypt data in our , (l. 125–126). Here’s a snippet that shows its (paginate your rows or your server will start sweating). subscribers backoffice demo page very basic usage <td>
  <span =" " - =" " =" :<? $ [' '];?>">
    <? $ -> ($ [' ']);?>
  </ >
</ >
< >
  < =" " - =" " =" : <? $ [' '];?>">
    <? $ -> ($ [' ']);?>
  </ >
</ > class moreinfo data toggle tooltip title string stored in database php echo row email php echo dao dec row email span td td span class moreinfo data toggle tooltip title string stored in database php echo row phone php echo dao dec row phone span td Looking at script, we can notice that before saving a new subscriber to database, we save.php check if a user with the same email or phone has already subscribed $email_hash = $dao->getBlindIndex($email); ($dao->exists_email($email_hash)) {
  header( ); ();
} $phone_hash = $dao->getBlindIndex($phone); ($dao->exists_phone($phone_hash)) {
  header( ); ();
}

$new_subscriber = $dao->insert($first_name,$last_name,$email,$phone,$privacy,$source,$ip); (!$new_subscriber) {
  header( ); ();
} {
  header( .$new_subscriber); ();
} // check if mail already exists if "location: index.php?e=email_exists" exit // check if phone already exists if "location: index.php?e=phone_exists" exit if "location: error.php" exit else "location: thankyou.php?id=" exit Function in check the existence of the same email by simply checking if the same blind index exists in email_hash field of the table. For better performance, we set an index on this email_hash field. Same logic occurs for function exists_mail() dao.php exists_phone() { (! ($email)) { ;
	} {
		$query = ->db->prepare( );
		$data = ($email);
		$query->execute($data);
		$all_mails = $query->fetchAll(); ($all_mails $row) { ;
		}
	} (PDOException $e) { ->logger->writeDbError($e->getMessage(), );
	} ;
} { (! ($phone)) { ; ();
	} {
		$query = ->db->prepare( );
		$data = ($phone);
		$query->execute($data);
		$all_numbers = $query->fetchAll(); ($all_numbers $row) { ;
		}
	} (PDOException $e) { ->logger->writeDbError($e->getMessage(), );
	} ;
} function exists_email ($email) if isset return true try $this "SELECT id  FROM subscribers WHERE email_hash=?" Array foreach as return true catch $this $this return false function exists_phone ($phone) if isset return false exit try $this "SELECT id FROM subscribers WHERE phone_hash=?" Array foreach as return true catch $this $this return false Once this function is clear, it is pretty straightforward to implement a in our demo backoffice page. See function in for implementation details. Obviously, the search will give you results only if the full email or full phone number is inputed, while are allowed. basic search by email or phone number listItems() dao.php no partial searches Final thoughts and warnings This article aims at showing how little effort is needed to meaningfully strengthen the privacy of your data in small / medium sized projects. The prototype described here must be taken as an opportunity to deepen our knowledge of encryption techniques and make it accessible to unexperienced developers and teams. As stated before, do not use this prototype in production environment, but play with it as much as you want.

How To Store Encrypted Data Collected By Your Web Application With PHP7 and LibSodium

Decentralization is the new fad . We need to remind ourselves why centralisation was good in the past. Centralisation was required in the past for efficient communication and information organisation. ‘Bad’ Data Problem Information has a tendency to spread like bacteria. But there is ‘good’ bacteria; then there is ‘bad’ bacteria. Centralised ‘control’ is ultimately about avoiding and restraining chaos. Centralised societies and systems, therefore, have vastly reduced levels of confusion, chaos and waste. A centralised system requires ‘good’ information in order to make its decisions. Because of this, centralised systems have extremely good ‘detection’ systems for ‘bad’ data. The of data contained within a centralised system, is critical to the system’s viability and existence. fidelity As a result, centralised systems are also data systems. all reliable ‘Information’ Dynamics To call centralised systems ‘reliable’ data systems is NOT the same thing as to claim that they have ‘more’ data than decentralised ones. In fact, centralised systems are poor at information . They tend to on (originally generated from sources) for information. extremely generation over-rely past data decentralised It is this that makes centralised systems extremely vulnerable to black swans; e.g. a change in central ‘rule’, a new ‘bacteria’ not previously factored in, a new product that changes how business is done, a deadly attack centralised system, etc. over-reliance on past data from another The dynamic of centralised systems then, is that of ‘waiting’ for centralised ‘command and control’. information The dynamic of decentralised systems has no ‘waiting period’, it is instant, continuous, ubiquitous and multi-dimensional. information Assessing System Risks It is incumbent upon every system engineer to know the risks behind centralised ‘rule’ and decentralized systems. both In the past, centralised rule systems seem to have much better than decentralised ones due to the fact that ‘information’ was to come by. Viewed another way, decentralised systems no real benefit _while c_arrying enormous risks of chaos and ‘ ! fared primarily hard offered bad’ information overload In the present, there is a predilection for decentralization. This is, partly, for reasons. Centralised systems carry the heavy risk of , being-disrupted, lacking-adaptability, and over-relying on history as opposed to the present and the future. reigning good stagnancy Systems Are About Information, Not ‘Control’ or Lack Thereof There is a way to judge ANY system, centralized or decentralized, on whether it is GOOD or not. This way, involves looking at the design process from the perspective of the system has 1) an generation, and 2) a system. new whether information ‘reliability’ detection A system that 1) is fragile and emergent properties of growth, adaptation and . lacks intelligence A systems that 2) is for complex processes. ineffective completing RETHINK Decentralization systems are not about ‘control’ or lack thereof. There is absolutely NO NEED to think of ‘decentralization’ as a paradigm to solving problems we have in the present. Since But we can ADAPT the concepts of decentralization in terms of how they apply to (see above section). information dynamics Having done that, we could then ask if our system has checks and balances for the of as and when required. Most ‘decentralization’ advocates like Nassim Taleb and Bitcoin/blockchain enthusiasts seem to miss this (extremely critical) point! new built-in execution complex processes There is an increasingly about decentralization that continues to fail to account for the of cohesion, and fidelity. exhausting excitement needs architectural RETHINK Information!!! Information is . It is, , not even about data ‘newness’! There is obviously, . (Perhaps the main cause behind ‘fake news’/ ?). NOT about quantities sadly a tendency, for ‘data (generation) intensive’ systems to have a predilection for ‘new’ data over ‘old’ data click-bait Information is really about ‘actionability’. Centralised systems this very well. Unfortunately, cetralised systems make the mistake of (inadvertently) then ‘limiting’ by consciously limiting the of information. know action amount System Design IS Information Architecture Many software engineers are usually taught that some form of ‘information artechitecture/design’ is required to design a complex piece of software. It is my contention in this essay that ANY system (physical, social/political or virtual) in fact an information architecture. existing is This implies that the ‘design process’ that is followed is MORE DETERMINANT to the final system ‘architecture’ than the ‘system architecture’ that an engineer drew at the beginning stages of the project. Another implication of this hypothesis is that it is NOT ONLY software engineers who actually DO ‘information architecture’. Every engineer is essentially responsible for achieving the same goal: making a system that is and somehow ‘generative’ of new insights. useful The obsession with BOTH AI and blockchain seems to prioritise ‘data’ over usefulness; ‘decentralization’ over centralisation; more ‘data’ over (actionable) data. present less But, we need BOTH! The TRUE Future ! The future lies in understanding this new ‘information’ paradigm. We need useful systems; we need information systems. truly also TO get decentralization is the WRONG paradigm for that to be realised. both, TO get , we need ‘more’ INFORMATION systems. (not necessarily ‘more data’). Thankfully, the Internet, blockchain, computing devices like cellphones, transportation systems (e.g. hyperloop, etc), social media and the proliferation of sensor devices, networks and technologies are of in this regard. both improved great help TO get we would need to several ‘check and balance’ that allow us to make our decisions and effectively. both, build systems own time-ously

Why Centralisation WAS A Good Thing

For better syntax highlighting and the ability to copy paste the code snippets, view this post on . grimoire.science One of the prob­lems of a shared sys­tem, is that some­times mul­ti­ple ap­pli­ca­tions are used by mul­ti­ple peo­ple. Normally this would be solved by the ex­cel­lent multi-user sup­port in­her­ent to *nix sys­tems. After all_,_ *nix and de­riv­a­tives were used by mul­ti­ple users at the same time. de­signed to be However, in some cases, it just makes sense for a sin­gle ap­pli­ca­tion to stay locked un­til a pass­word is sup­plied. In any case, is one of the bet­ter en­cryp­tion meth­ods (as long as us­abil­ity over­shad­ows se­cu­rity) and it’s a good prac­tice to en­crypt ap­pli­ca­tion data any­way. The case for en­cryp­tion varies, but weather it’s to make life harder for hack­ers, or just to stop cloud stor­age providers from sniff­ing around, it’s always a good idea. Even if mul­ti­ple users will not be shar­ing an ac­count, en­cryp­tion pre­vents the or from get­ting too frisky with your data. encfs root users admins The pop­u­lar and other se­cu­rity au­then­ti­ca­tion meth­ods are not a good ﬁt for this since most of them are un­locked all at once and with­out be­ing linked to a par­tic­u­lar pro­gram. gnome-keyring Here we will cre­ate a few sim­ple bash scripts to gener­i­cally and ﬂex­i­bly lock and en­crypt ap­pli­ca­tions in a way which al­lows for mul­ti­ple users to have their own pri­vate en­crypted in­stances of shared apps. The com­plete scripts are lo­cated in my . Dotﬁles Requirements The ba­sic re­quire­ments are com­monly found across most UNIX sys­tems and its de­riv­a­tives in­clud­ing MacOS, Ubuntu and other Debian dis­tros, RPM based sys­tems etc. I per­son­ally run but that’s just a bi­ased en­dorse­ment. Arch Linux The re­quire­ments are: Bash This is needed for the shell sub­sti­tu­tions in the shell script. Found al­most every­where. 2. EncFS This han­dles the en­cryp­tion por­tion. Though every­thing cov­ered here will re­quire only the base pro­gram it­self, new users would prob­a­bly ben­e­ﬁt from hav­ing one of the GUI in­ter­faces to as well. I pre­fer . encfs Gnome Encfs Manager 3. An ASKPASS pro­gram These are most fa­mously known for weird er­rors. However, here we will fo­cus on and as a fall­back. ssh zen­ity git And that’s it. Other vari­a­tions of this method might use for more GUI good­ness. Other pro­grams and helpers may also be used, like the pop­u­lar pro­gram. shoes askpass x11-ssh-askpass Program Structure Our basic structure is simple. An en­crypted folder is mounted The ap­pli­ca­tion is run Additionally we would like the fol­low­ing fea­tures: Execution with­out the ter­mi­nal (GUI, no ter­mi­nal user queries) An au­to­mated way of gen­er­at­ing a new stash for ap­pli­ca­tions A san­i­tized name for the mount points Preliminaries Before get­ting to the cre­ation of a script, I like to ex­per­i­ment with the na­tive shell. In this case this sim­ply in­volved check­ing the fol­low­ing: Shell trials This prompted me to cre­ate the di­rec­tory if it did­n’t ex­ist, which would not be han­dled prop­erly from within a shell script. Additionally the MAN page for showed me that sup­port for ex­ter­nal au­then­ti­ca­tion man­agers is granted via the ﬂag. encfs --extpass Implementation With those pre­lim­i­nar­ies out of the way, it is time to start script­ing. Portions which re­quire the bash shell specif­i­cally will have the she­bang in­cluded. Always re­mem­ber to start the script with it and to only put it once, right at the top of the ﬁle. The shebang Setting Variables Initially we might simply set an unlock string as follows: Using a variable for the unlock dialog Choosing an ASKPASS pro­gram Since scripts can quickly get clunky with­out in­tend­ing too, we will ﬁrst add a sim­ple vari­able which is suit­able for run­ning the ex­ter­nal au­then­ti­ca­tion. Zenity Prompt Zenity implementation As men­tioned pre­vi­ously, is the pret­tier choice, how­ever, it may not be in­stalled every­where. So we need a fall­back. zen­ity is more or less avail­able every­where, and it just so hap­pens to have a pretty neat tool as well. Git askpass Git Askpass Prompt Git Askpass Implementaion However, it would be bet­ter to wrap them both up in a way to pick one or the other based on the avail­abil­ity. So, we write a sim­ple test. The test logic Honestly the us­age of in­stead of is a bit con­tro­ver­sial. However, here I went with sim­ply be­cause it seemed faster. The more portable (POSIX com­pli­ant) ver­sion of the above would use . For more de­tails check . which command -v which command -v this stack ex­change ques­tion Creating Mount-points de­faults to re­mov­ing the mount point when the stash is un­mounted, how­ever, this causes a ter­mi­nal in­put de­mand which needed to be sup­pressed, hence the di­rec­to­ries are cre­ated prior to run­ning Encfs. Gnome Encfs Manager Variables and directories Caveats The above snip­pet does not deal with sit­u­a­tions where: The stash is al­ready mounted The stash does not ex­ist These are dealt with in the sec­tion of this doc­u­ment. Improvements Mount and Run Now we are in a po­si­tion to sim­ply mount our stash and run the pro­gram. A basic mount and run Caveats At this stage the script is not equipped to deal with sit­u­a­tions where: The mount op­er­a­tion fails (wrong pass­word) The con­ﬁg ﬁles are en­crypted The script runs the pro­gram with­out test­ing the re­sult of the mount, which will lead to much frus­tra­tion and weird er­rors. These edge cases are handled by the code in . Handling Authentication Improvements Several im­prove­ments to the ba­sic script cre­ated above are dis­cussed in this sec­tion. Naming di­rec­to­ries This is ac­tu­ally not a re­ally im­por­tant bit, how­ever, I wanted the app di­rec­to­ries to start with let­ters. Also I wanted the en­crypted data to be stored in a folder. cap­i­tal hid­den In any case, this por­tion of the script uses a bash spe­ciﬁc ex­pan­sion. At thispoint we can also make the a little neater. unlockString Bash substitutions for prettier names Now that we have the name, we simply modify the directories. Renaming the directories Handling mounted di­rec­to­ries To en­sure that the script is able to even­tu­ally deal with sit­u­a­tions where the com­mand is run in suc­ces­sion, a check is re­quired to ﬁg­ure out if the mount point is cur­rently mounted. If it is mounted, we will un­mount it. Testing for mounts Additionally, for cases where the stash does not yet ex­ist, we will need to cre­ate the other di­rec­tory as well. We shall also kill the ap­pli­ca­tion if the stash is to be mounted (for se­cu­rity). This por­tion was aided by . this stack ex­change thread Mount handling with directory creation Handling Authentication Finally we shall deal with cases where the script ex­e­cutes and the stashes ex­ist, but the pass­word is in­cor­rect. Additionally, we shall deal with man­ag­ing the ﬂow of con­trol via the vari­able. $? Quite sim­ply, the vari­able holds the re­sult of the pre­vi­ous com­mand. Hence it can be used to con­trol the ﬂow. This was in­spired by the an­swers . $? here The results variable and flow of control Putting it all to­gether For the lat­est re­vi­sions check . my Dotﬁles It is also re­pro­duced here as a gist, since Medium can’t handle syntax highlighting. Future Directions There ought to be a non-ter­mi­nal way of cre­at­ing the stash for the ﬁrst time. Also, it may be in­ter­est­ing to work on the rules and con­ﬁg­u­ra­tion schemes for a va­ri­ety of ap­pli­ca­tions. Originally published at grimoire.science .

Locking and Encrypting Apps with Encfs

OK, I understand that it is kind of funny to have a monitor to your monitors and no, this is not a halloween trick. The use case here is something that maybe a lot of you out there had faced before like I did. Let’s think for a second about a simple, common scenarios that all we IT guys deal with in a daily basis. You have your infrastructure, your applications and you fine monitoring which gather metrics from all sorts of components and aggregates it in CloudWatch (or any other place) so you can joyfully see all those colorful charts. One day your client calls you and informs you that he cannot use your services due to an issue with your API or Website. You start wondering what is happening as you “didn’t receive any alerts”; by the time you get to your awesome metrics you see that all of them are empty because the application that feeds it to CloudWatch has DIED! #dramaticbeaver Here at my work we have such a setup and, of course, we had felt this pain and, of course, it was on a weekend and, of course, it was past 10 PM and, of course, we were only alerted by a client who, of course, was having timeouts. #opslife In our environment we have a lot of micro services, backend and user facing applications that all outputs data that we grab and ship to CloudWatch so we can build metrics, charts and get alerted by them. This is an easy to build AWS managed setup that suits us really well and help us troubleshoot all sorts of issues. The problem is that our software that was grouping and shipping those metrics is not bug free. As any other software it can fail and it WILL FAIL. The first idea any IT profession will come up with is: “LET’S MONITOR IT!”. Ok, this is nice, but if you think for a second you are creating a monitor for a monitor, and this new monitor can also fails, so maybe in the future you will need a new monitor to monitor the monitor for your monitors. But hey! This new monitor can also fail so maybe we should also have a monitor to monitor the new monitor that is monitoring the monitor of your monitor — I think you know where I am going with this, right? So, we need a new monitor. Period. But one that we can trust like 99% (I’LL NEVER SAY THIS IS 100% GUARANTEED) of the time and that we will not end up falling on the infinite loop of monitors. You have your monitor or metric shipper, call it whatever you want, and you need to make sure all the metrics you push to CloudWatch are actually there. Thank Bezos, AWS has an awesome API to all their services and CloudWatch is not an exception so, let’s build a Lambda function that will simply read the metrics you have and in case it cannot find the metric it will alert you. In this example I am generating an PagerDuty alert but you can change it for whatever tool you have. For example, you can push Slack notification as well. you may find the lambda code. Here in this gist An overview of the code: : will iterate over our metrics, try to read the metric from AWS CloudWatch service and count the total number of metrics that it had errors during this process. If the total amount it greater than zero then it will generate an incident alert; main() : this one will basically create the incident for us; pagerduty() : the AWS Lambda service will need to call a method inside your code and it will need a function that receives two parameters, the event and the context, we are simply ignoring most of it and only using the Event ID to output that to our logs; lambda_handler() In order to create the lambda you will need to generate a zip code containing not only the code above but also the libraries you will. In this case we are only using some core libraries and the Boto3 package that you need to install in the same directory of your code. Refer to in order to do that in your computer, the procedure to do that will vary depending on your OS. this guide here Creating the actual lambda function is not the scope of this article but you can use this document in order to do that. Basically, what you need is: A zip file containing the source code of the lambda that you created following the steps provided in the above link; You will need to create a so you can use as a scheduled trigger to your lambda (I am running my link a cron job every 5 minutes); CloudWatch Event Rule An that your lambda function will use and this role must have read access to CloudWatch; IAM Role Now you have a cost effective, simple but trustworthy solution to monitor your monitors and alert you before your customer does!

Using AWS Lambda to Escape the Monitor of Monitors Infinite Loop

“If you’re not paying for a service, you’re not the customer — you’re the product.” — Steve Ballmer Former Microsoft CEO (and now owner of the LA Clippers) Steve Ballmer expresses a fundamental truth about media. And, as I’ll illustrate in a second, he also reveals why, even at the price of one billion dollars, the LA Clippers is a great deal for the discerning multi-billionaire shopping for a major sports franchise. With this G5 at his disposal, Steve Ballmer never has to miss an away game. Once upon a time, a large and profitable industry was created by the new technologies of radio and then television. It was based on a very simple business transaction: broadcasters attracted viewers with free and then they sold the of their users to advertisers. The consumer literally pays for the programming with their attention, by watching ads. There was no other cost. The broadcast was absolutely free, like the air we breathe. There were only a few channels, so the broadcasters had scarcity on their side, too. Other than keeping an appointment with your television, there was no way other to see a program ever again. The three major networks formed a “tri-opoly”. They were television. programming attention Had my mother not placed me in this position for several hours a day (more on weekends) I might never have had a career in entertainment. The introduction of cable television, and the elimination (by law) of the actual over-the-air broadcasts, has forced people to pay for what was once free. Instead of three channels, now there are three hundred. Of course, the increased choice in programming has a price. A hefty price. We’re now paying $100 a month for access to a service that was once free, and there are still commercials. Let me repeat that in case you were listening: we’re now paying over $1200 a year per household for a service that was once free. Some crazy math: there are 117.5 million households in the U.S. That’s 141 billion dollars. American families now spend more on cable television than they do on medical insurance. The broadcasters whose channels the cable company carries, all collect a license fee from the cable company, so everyone gets a taste of the $1200, which is how good monopolies (and gangster economies) work. The more people who are profiting from corruption, the more normal and accepted it becomes, and therefore more difficult it is to eliminate. When you’re skimming off the top of one of the world’s largest economies you don’t need to bother with a sports team. Here’s the world’s richest man congratulating Russian wrestler Fedor Emelianenko (bare-chested) on his win over U.S. fighter Jeff Monson in Moscow. The broadcasters have come through this disruption rather well. They get a fee from the cable companies for doing what they used to do for free and they still have advertisers paying for their customers’ attention. They quickly figured out that by having a bunch of different channels they could get more fees, and offer advertisers better targeted audiences. So channels were created and/or purchased (see below) all bankrolled by subscriber fees paid to broadcasters, cable companies, and rich entrepreneurs like Rupert Murdoch and Ted Turner, who both leveraged their family publishing empires into broadcast media. Yes, TV used to be free, but for $1200/year look how many channels you get. Advertisers, on the other hand, have actually been disrupted in a multitude of negative ways. Time shifting — watching taped shows — has reached epidemic proportions. Hypertargeting an audience isn’t valuable if the audience time shifts the programming. While digital media has vastly increased the options open to advertisers, there is no good way to replace those lost millions of eyeballs using youtube or social media. It’s very hard to get to a million a couple of hundred at a time. If you want to sell hundreds of thousands of cars this quarter, your media plan is all about television, not social media. There is still one spot where time shifting doesn’t work, where the ads are anticipated, where an appointment still must be kept: major sporting events like the Super Bowl. Hence the 9MM per minute cost to advertisers. A sports fan has to watch sports event programs in real time. Other sporting events are similarly in demand. Buying the LA Clippers is a slam dunk. Sports are not time shifted and therefore command a premium from advertisers. And the leagues command a premium to those channels that carry their games, so much that teams now get their own channels. They can even charge additional premiums for restricted games and internet access, as the NBA, Baseball and Football now do. Ballmer’s not just buying a team, he’s buying a channel. And when you’ve got one channel, you’re in line to get another. And when you’ve got one sports team… You’d never know these cars are moving at 60 mph. Better leave plenty of time to get to the game. There is no mass transit solution. Two other important things to note. First, that the NBA is the fastest growing American sports league, with huge international interest. Second, Ballmer’s buying into Los Angeles itself. LA is a wealthy, progressive, multi-ethnic mixing bowl where more than 10% of the country lives. The traffic’s getting worse (there are too many damn people there) but the air is getting better, and the sunsets are consistently breathtaking. Barring an epic earthquake, fire and/or drought, LA is a pretty amazing place to own a basketball team, or any business. Marketers must fish where the fish are. Enter Steve Ballmer bearing a check for a cool billion. He’ll get it back and then some. Six companies pretty much own all the media in the US.

Pay Money or Pay Attention

A VPN, or Virtual Private Network, is a service that protects your online privacy by connecting your device to the Internet through an “encrypted tunnel. While this may sound simple, in reality encryption is notoriously difficult--and misunderstood. As a result, many misconceptions have sprung up about VPNs, leading to faulty and distorted ideas about what they are--and aren’t.In this article, we’ll address the five top myths about VPNs, explain why they’re wrong, and highlight how VPNs can make your online life safer and easier. Myth 1. VPNs are Illegal Since VPNs obscure your identity, they often conjure up images of hackers, online pirates stealing new blockbuster movies, and the “dark web.” Over time, this has created some speculation about their  legitimacy. Reality: Certain countries do restrict the use of VPNs, or consider them illegal. Currently, this list includes Belarus, China, Iran, Iraq, North Korea, Oman, Russia, Turkey, Turkmenistan, and the UAE. Depending on the country, governments may impose fines on both the VPN user and the VPN service provider for using an unsanctioned VPN. For example, in the UAE, a VPN is legal--but if it’s used to commit a crime, the penalties for that crime may be increased as a result. There are numerous reasons why companies large and small--and even governments--use VPNs, such as creating a for remote work. As a result, most countries accept VPNs as a legal service. protected corporate network It’s always a good idea to where you intend to use a VPN before trying to connect. And, most importantly: While a VPN may be legal, cybercrime is not! check the regulations of the specific country Myth 2. If you use a VPN, the government is watching you In the late 1950s, five countries joined efforts to share their intelligence data in the . Over the course of time, the alliance’s role has extended to the surveillance of citizens’ online activity and has grown to nine, and later to fourteen, countries: Five Eyes Alliance Five Eyes: US, UK, Canada, Australia, New Zealand Nine Eyes: Five Eyes + Denmark, France, Holland, Norway Fourteen Eyes: Nine Eyes + Germany, Belgium, Italy, Sweden, Spain A common myth about choosing a VPN provider is that it’s best to avoid one that’s based in a Fourteen Eyes member state, because the provider may be forced to disclose your information to the government. Another common myth is that VPN providers with  “offshore” servers can refuse to comply with governments’ requests for information. Reality: Though these concerns about surveillance can be valid, the link between where your VPN provider is located and your chances of being under surveillance isn’t straightforward. First of all, the number of countries that have been caught or suspected of exchanging intelligence information goes far beyond the “Fourteen Eyes” list of members. There’s ultimately no way to verify which countries disclose information to meet the requests of other, usually more powerful, states. Second, VPN providers may need to comply with the laws of multiple countries: The country where they’re registered, the country where their servers are located, and also the country where they operate. This doesn’t necessarily mean that a provider must comply with every request for information from a government--but it’s important to check the regulations that apply to your country, so you have a clear picture of what can be expected. Regulations can also be a double-edged sword, which can help VPN users: Human rights groups have been known to successfully , protecting VPN use. push back on unconstitutional information gathering Myth 3. VPN providers collect data about their users VPN providers have access to a vast pool of sensitive data, and as a result many people have come to fear  that the providers may sell it to the highest bidder. This information can include the websites you visit, your shopping habits, your voting preferences, and even your internet connection logs, which can show your location. Reality: While certain VPN providers--usually those who offer services for free--collect user data for their own gain, most VPN providers abide by a “no-log policy”. They collect only basic  user details, like source IP addresses, VPN IP addresses, and connection start and stop timestamps, so they can offer a stable connection. These details are logged for a limited time and are removed immediately upon disconnection from the VPN. Connection logs are stored for a certain period of time, according to specific legal requirements, and then are erased from the database. Myth 4. A VPN protects you from cyberattacks This is probably the most dangerous misconception about how a VPN works: Users assume that anonymity makes you unrecognizable by hackers, and safe from attacks as a result. Reality:  It’s true that the 256-bit encryption grants you near-total anonymity and hides you from most hackers. In this sense, a VPN offers solid protection for your privacy and security online. But no matter how powerful that encryption is, it does not protect you from cyberattackss if you willfully share your data by, for example, clicking a dubious link. A VPN also isn’t a substitute for antivirus software. For this reason, advanced VPN services are delivered as a package that includes additional malware protection options, such as a to block known malware websites, or to prevent the routing of your traffic outside the VPN. DNS firewall IPv6 leak protection Whatever kind of Internet connection you use--whether it’s open or VPN-protected--you should stay vigilant so you don’t fall victim to hacking, phishing or viruses. Myth 5. A VPN always slows down your connection This myth comes from the fact that VPNs rely on data encryption, and transmit protected data through a “VPN tunnel.” In this process, each data packet is enclosed in a separate encrypted outer packet, transmitted to the server, and decrypted so that the data inside can be accessed. This can be quite time-consuming and may slow down the internet connection. Reality: A slower connection may be the case if the server is located on the other side of the world. The best option is to choose a server that’s located near the place where you’re using the VPN--unless you specifically want to use a VPN to mask your location (for example, to access content that’s blocked in your own country). Another solution is to opt for a paid VPN, which gives you priority server access over non-paying users. By taking these steps, you can find a VPN that maintains speeds similar to the open Internet. If you use careful switching and avoidISP throttling, you can even increase your VPN’s speed. When you’re using a VPN, you may notice  a lower speed on some sites (as compared to an open connection) and a much faster speed on other sites or services. This discrepancy happens when Internet Service Providers reduce the speed of some traffic types, like peer-to-peer traffic. But a VPN can usually solve this problem by hiding the type of data transmitted inside an encrypted signal--and preventing the Internet Service Provider from slowing the traffic down. Why Getting The Real Facts About VPNs Is Important This article is just the tip of the iceberg when it comes to VPNs: The more research you do, the more myths – and facts – you’ll discover. By sharing these top five myths here, and the reasons why they’re untrue, we’ve shown how a VPN is a legal, safe and powerful tool--provided that you take precautions and use it only for legal purposes.

Stories, News & Mentions about OPSLY

Opsly’s Stories on HackerNoon