Akto is an instant API security platform. We are open source. You can self-host Akto or use our cloud version, both free. Github: https://github.com/akto-api-security/akto 1️⃣ Don't have any visibility into your internal, public and partner APIs? 2️⃣ Don't know what all APIs are leaking PII data? 3️⃣ Don't know if your APIs are vulnerable, have broken authentication and authorization? 4️⃣ Don't know how to test for business logic vulnerabilities in APIs? Welcome to Akto !🚀 It's the world's first proactive API Security platform.

Akto

akto

This paper is available on arxiv under CC 4.0 license. Authors: (1) Ayei E. Ibor, Trustworthy Digital Infrastructure for Identity Systems, The Alan Turing Institute, United Kingdom; (2) Mark Hooper, Trustworthy Digital Infrastructure for Identity Systems, The Alan Turing Institute, United Kingdom; (3) Carsten Maple, Trustworthy Digital Infrastructure for Identity Systems, The Alan Turing Institute, United Kingdom; (4) Gregory Epiphaniou, WMG, University of Warwick, United Kingdom. Table of Links Abstract & Introduction Background Review of Literature Methodology Discussion of Findings Conclusion and Future Work & References 6. Conclusion and Future Work Trustworthy interoperability is a key enabler for identity management and widens the dimensions of e-government in developing countries. This research provided insights into the challenges, opportunities, and requirements of interoperability and highlighted the limitations of current interoperability solutions such as the Estonian X-Road and the eIDAS of the European Union. We found that each existing interoperability solution has its benefits and drawbacks, although both are targeted at a common goal. To achieve seamless communication of identification data, we proposed an approach for scoping requirements by first identifying the key processes to identity management in an interoperability context. Furthermore, we addressed the limitations of the current solutions using an architecture that considers the flexibility of identity verification and validation through an interoperable layer that is based on four interacting layers of data representation, semantics, binding, and security and privacy. For future work, an analysis of country-specific use cases for achieving interoperability in the African identity ecosystem will be performed. References African Union, 2020. The Digital Transformation Strategy for Africa (2020-30), Available at https://openresearch-repository.anu.edu.au/bitstream/1885/278587/1/GhaEcon_24.pdf, Accessed 20 May 2023. Alamillo, I., Mouille, S., Rock, A., Soumelidis, N., Tabor, M., Gorniak, S., 2023. Digital identity standards: Analysis of standardisation requirements in support of cybersecurity policy, Available at file:///C:/Users/aIbor/Downloads/Digital_Identity_Standards%20(1).pdf, Accessed 30 July 2023. Aldosary, M. and Alqahtani, N., 2021. A Survey on Federated Identity Management Systems Limitation and Solutions. International Journal of Network Security & Its Applications (IJNSA) Vol, 13. ANSSI, 2023. The EIDAS Regulation, Available at https://www.ssi.gouv.fr/en/regulation/digital-confidence/the-eidas-regulation/, Accessed 09 April 2023. Atick, J., 2016. Digital identity: the essential guide. In ID4Africa Identity Forum (Vol. 2016, pp. 1-3). Backhouse, J. and Halperin, R., 2009. Approaching interoperability for identity management systems. In The Future of Identity in the Information Society (pp. 245-268). Berlin, Heidelberg: Springer Berlin Heidelberg. Bakhtina, M., Matulevičius, R., Awad, A. and Kivimäki, P., 2022. Rebooting Trust Management in X-Road, Public Report. Nordic Institute for Interoperability Solutions (NIIS). Bhattarai, R., Pappel, I., Vainsalu, H., Yahia, S.B. and Draheim, D., 2019. The impact of the single digital gateway regulation from the citizens’ perspective. Procedia Computer Science, 164, pp.159-167. Bandura, R. and Ramanujam, S.R., 2021. Developing inclusive digital payment systems. Center for Strategic and International Studies, 21. Benaddi, H., Laaz, N., Bouhlal, A. and El Kettani, E., 2023. Data storage architecture for egovernment interoperability: Morocco case. Indonesian Journal of Electrical Engineering and Computer Science, 29(3), pp.1678-1686. Catuogno, L. and Galdi, C., 2014. Achieving interoperability between federated identity management systems: A case of study. Journal of High Speed Networks, 20(4), pp.209-221. Colbern, A. and Ramakrishnan, S.K., 2020. Citizenship reimagined: A new framework for state rights in the United States. Cambridge University Press. Cuijpers, C.M.K.C. and Schroers, J., 2014. eIDAS as guideline for the development of a pan European eID framework in FutureID. Domingo, E. and Teevan, C., 2022. Africa’s journey towards an integrated digital payments landscape and how the EU can support it. ECDPM brief, 23. European Commission, 2023a. X-Road Data Exchange Layer, Available at https://joinup.ec.europa.eu/collection/ict-security/solution/x-road-data-exchange-layer/about, Accessed 10 April 2023. European Commission, 2023b. Shaping Europe’s digital future, Available at https://digitalstrategy.ec.europa.eu/en/policies/discover-eidas, Accessed 20 April 2023. Fathalla, E.S., Azab, M., Xin, C. and Wu, H., 2023. PT-SSIM: A Proactive, Trustworthy SelfSovereign Identity Management System. IEEE Internet of Things Journal. Grassi, P.A., Garcia, M.E. and Fenton, J.L., 2017. Digital identity guidelines. NIST special publication, 800, pp.63-3. Gelb, A. and Metz, A.D., 2018. Identification revolution: Can digital ID be harnessed for development?. Brookings Institution Press. Halili, F. and Ramadani, E., 2018. Web services: a comparison of soap and rest services. Modern Applied Science, 12(3), p.175. Hoffmann, T. and Solarte-Vasquez, M.C., 2022. The estonian e-residency programme and its role beyond the country’s digital public sector ecosystem. Revista CES Derecho, 13(2), pp.184-204. Hölbl, M., Kežmah, B. and Kompara, M., 2023. eIDAS Interoperability and Cross-Border Compliance Issues. Mathematics, 11(2), p.430. Hummer, D. and Rebovich, D.J., 2023. Identity theft and financial loss. In Handbook on Crime and Technology (pp. 38-53). Edward Elgar Publishing. Irvin-Erickson, Y., 2023. How Does Immigration Status and Citizenship Affect Identity Theft Victimization Risk in the US? Insights from the 2018 National Crime Victimization Survey Identity Theft Supplement. Victims & Offenders, pp.1-24. Kaaniche, N., Laurent, M. and Belguith, S., 2020. Privacy enhancing technologies for solving the privacy-personalization paradox: Taxonomy and survey. Journal of Network and Computer Applications, 171, p.102807. Kanagwa, B., Nakatumba-Nabende, J., Mugwanya, R., Kigozi Kahiigi, E. and Ngabirano, S., 2018. Towards an interoperability e-government framework for Uganda. In e-Infrastructure and e-Services for Developing Countries: 9th International Conference, AFRICOMM 2017, Lagos, Nigeria, December 11-12, 2017, Proceedings 9 (pp. 16-28). Springer International Publishing. Kiourtis, A., Giannetsos, T., Menesidou, S.A., Mavrogiorgou, A., Symvoulidis, C., GRAZİANİ, A., KLEFTAKİS, S., MAVROGİORGOS, K., ZAFEİROPOULOS, N., GKOLİAS, C.A. and KYRİAZİS, D., 2023. Identity management standards: A literature review. Computers and Informatics, 3(1), pp.35-46. Krimmer, R., Dedovic, S., Schmidt, C. and Corici, A.A., 2021, August. Developing crossborder e-Governance: Exploring interoperability and cross-border integration. In International Conference on Electronic Participation (pp. 107-124). Cham: Springer International Publishing. Kotzé, P. and Alberts, R., 2017. Towards a Conceptual Model for an e-Government Interoperability Framework for South Africa. In ICEIS (3) (pp. 493-506). Lips, S., Bharosa, N. and Draheim, D., 2020. eIDAS implementation challenges: the case of Estonia and the Netherlands. In Electronic Governance and Open Society: Challenges in Eurasia: 7th International Conference, EGOSE 2020, St. Petersburg, Russia, November 18– 19, 2020, Proceedings 7 (pp. 75-89). Springer International Publishing. Luong, D.A. and Park, J.H., 2023. Privacy-Preserving Identity Management System on Blockchain Using Zk-SNARK. IEEE Access, 11, pp.1840-1853. Manda, M. I. and Backhouse, J., "Addressing trust, security and privacy concerns in egovernment integration, interoperability and information sharing through policy: a case of South Africa" (2016). CONF-IRM 2016 Proceedings. Paper 67. http://aisel.aisnet.org/confirm2016/6 Masiero, S., 2023. Digital identity as platform-mediated surveillance. Big Data & Society, 10(1), p.20539517221135176. McBride, K., Kütt, A., Ben Yahia, S. and Draheim, D., 2019, November. On positive feedback loops in digital government architecture. In Proceedings of the 11th International Conference on Management of Digital EcoSystems (pp. 174-180). Mocanu, S., Chiriac, A.M., Popa, C., Dobrescu, R. and Saru, D., 2019. Identification and trust techniques compatible with eIDAS regulation. In Security and Privacy in New Computing Environments: Second EAI International Conference, SPNCE 2019, Tianjin, China, April 13– 14, 2019, Proceedings 2 (pp. 656-665). Springer International Publishing. Pöhn, D. and Hommel, W., 2023. Towards an Improved Taxonomy of Attacks Related to Digital Identities and Identity Management Systems. Security and Communication Networks, 2023. Priisalu, J. and Ottis, R., 2017. Personal control of privacy and data: Estonian experience. Health and technology, 7(4), pp.441-451. Saputro, R., Pappel, I., Vainsalu, H., Lips, S. and Draheim, D., 2020, April. Prerequisites for the adoption of the X-Road interoperability and data exchange framework: a comparative study. In 2020 Seventh International Conference on eDemocracy & eGovernment (ICEDEG) (pp. 216-222). IEEE. Schmidt, C. and Krimmer, R., 2022. How to implement the European digital single market: identifying the catalyst for digital transformation. Journal of European Integration, 44(1), pp.59-80. Sedlmeir, J., Smethurst, R., Rieger, A. and Fridgen, G., 2021. Digital identities and verifiable credentials. Business & Information Systems Engineering, 63(5), pp.603-613. Sharif, A., Ranzi, M., Carbone, R., Sciarretta, G. and Ranise, S., 2022, August. SoK: A Survey on Technological Trends for (pre) Notified eIDAS Electronic Identity Schemes. In Proceedings of the 17th International Conference on Availability, Reliability and Security (pp. 1-10). Sharma, R. and Panigrahi, P.K., 2015. Developing a roadmap for planning and implementation of interoperability capability in e-government. Transforming Government: People, Process and Policy, 9(4), pp.426-447. Solvak, M., Unt, T., Rozgonjuk, D., Võrk, A., Veskimäe, M. and Vassil, K., 2019. Egovernance diffusion: Population level e-service adoption rates and usage patterns. Telematics and Informatics, 36, pp.39-54. Spiliotopoulos, T., Sheik, A.T., Gottardello, D. and Dover, R., 2023. Onboarding Citizens to Digital Identity Systems. arXiv preprint arXiv:2306.13511. Srinivas, J., Das, A.K. and Kumar, N., 2019. Government regulations in cyber security: Framework, standards and recommendations. Future generation computer systems, 92, pp.178-188. Tang, F., Ma, C. and Cheng, K., 2023. Privacy-preserving authentication scheme based on zero trust architecture. Digital Communications and Networks. The World Bank, 2023a. ID4D Global Dataset, Available at https://id4d.worldbank.org/global-dataset, Accessed: 17 July 2023. The World Bank, 2023b. Practitioner’s Guide: Interoperability, Available at https://id4d.worldbank.org/guide/interoperability, Accessed: 22 April 2023. Walters, R., 2023. Theft [Identity]. In Cybersecurity and Data Laws of the Commonwealth: International Trade, Investment and Arbitration (pp. 297-306). Singapore: Springer Nature Singapore. World Bank Group, 2016. ID4D Country Diagnostic: Nigeria, 2016, Available at https://documents1.worldbank.org/curated/en/136541489666581589/pdf/113567-REPLNigeria-ID4D-Diagnostics-Web.pdf, Accessed 20 May 2023. W3C Recommendation, 2022. Verifiable credentials data model v1.1, Available at https://www.w3.org/TR/vc-data-model/#dfn-credential, Accessed 04 April 2023. World Bank Group, 2018. Guidelines for ID4D Diagnostics, Available at https://documents1.worldbank.org/curated/en/370121518449921710/Guidelines-for-ID4DDiagnostics.pdf, Accessed: 22 April 2023. X-Road, 2020. X-Road: Message Transport Protocol, Available at https://www.xtee.ee/docs/live/xroad/pr-messtransp_x-road_message_transport_protocol.html, Accessed 12 April 2023. X-Road, 2023a. X-Road Security, Available at https://x-road.global/security, Accessed 03 April 2023. X-Road, 2023b. X-Road Technology Overview, Available at https://x-road.global/x-roadtechnology-overview, Accessed 10 March 2023. X-Road, 2023c. X-Road: Use Case Model for Member Communication, Available at https://docs.x-road.global/UseCases/uc-mess_x-road_member_communication_use_case_model.html#version-history, Accessed 12 April 2023. X-Road, 2023d. Technical Specification: X-Road Architecture, Available at https://docs.xroad.global/Architecture/arc-g_x-road_arhitecture.html#version-history, Accessed 12 April 2023. Yin, J., Xiao, Y., Pei, Q., Ju, Y., Liu, L., Xiao, M. and Wu, C., 2022. SmartDID: a novel privacy-preserving identity based on blockchain for IoT. IEEE Internet of Things Journal, 10(8), pp.6718-6732. This paper is available on under CC 4.0 license. Arxiv

Trustworthy Interoperability and Identity Management Systems: What Comes Next

As part of the series, this is my story. Tech Like Us I was born just outside of Mumbai, India. My family emigrated to the U.S. when I was 3 years old, and I grew up next to cornfields in a sleepy Ohio town. So while my ethnic background is Indian, I identify as an American from the Midwest. I refer to soda as “ ” and am prone to altitude sickness at the slightest sight of a hill. correctly pop I got interested in tech as a kid tinkering with all the electronic “toys” my dad would bring home like mobile phones the size of briefcases, hefty 10-pound camcorders, and computer towers that were as tall as me (and probably still are). I enjoyed ripping apart floppy disks to see what was inside and getting “under the hood” of our PCs to install some extra memory. And of course, I vividly remember the exciting early days of the Internet, including the anticipation of receiving a brightly colored in the mail. Once we were on the World Wide Web, creating webpages and learning how to code became my other favorite hobbies. AOL CD At the , I majored in Computer Engineering, which is an amalgam of Computer Science and Electrical Engineering. It blended software and hardware, along with theory and practice. I also minored in Mathematics to fully surrender to my nerdy tendencies. University of Michigan After undergrad, I stayed in the tech field but moved into a consulting role with . I visited clients all over the world, from Des Moines, IA to Anchorage, AK to Tokyo, Japan. At the end of my fourth year of consulting, I realized I wanted to equip my technical knowledge with a formal business education. Citrix Systems So I went to the University of Chicago’s where I concentrated in Finance and Entrepreneurship and worked in startups, both during and after. Business school was a fantastic experience for me. I had never even seen a balance sheet prior to grad school, but within those 2 years I pitched my own business plan to venture capitalists, took classes taught by Nobel Laureates, and met some of the most amazing over-achievers who are now some of my closest friends. Through that experience I also found my niche in tech startups and moved to the West Coast shortly after graduation. Booth School of Business Since moving out to the “Best Coast,” I’ve been working as a product manager at a broad range of tech companies. My first foray into the Silicon Valley scene was at a 7-person startup with Series A financing. Within a year, our company was acquired. After experiencing the ins and outs of an acquisition first-hand, I moved to a large company in , where I created a completely brand-new analytics-based product called . I am now in a lead product position at , where I’ve had opportunities to manage all of our product lines, develop strategies and goals across the business, build the product team, and see the company grow from 15 people to a team of 65 in the short time I’ve been here. Rackspace Cloud Intelligence PubNub I love the product manager role because it’s an exciting mix between the technical and business sides of product development. In this role, I’ve had opportunities to develop completely new tech products, drive conceptual ideas into real market offerings, create and influence company strategies, speak at industry events, and advise other startups and entrepreneurs. While I’m grateful for the enriching professional experiences I’ve had, looking back there were instances where being a woman in the field of tech made me feel like an outsider. For instance, it felt isolating to be one of only a handful of women in my major coursework in college and that trend persists in the professional realm. It was also frustrating when strangers spoke to me in a condescending manner or automatically assumed I was an assistant, instead of the conference speaker, product owner, or technical expert. But when you find yourself in those situations, stay confident and remember that you are an integral part to changing the makeup of the tech field! Your voice matters. Don’t shy away from having your opinions heard, don’t apologize for being awesome, and above all else — be yourself and let your contributions shine. I have also found such amazing people and camaraderie in the tech world, and know that you will too. I believe that those of us privileged enough to be a part of this field should tell our stories and mentor young women and men so that the future of our industry is more balanced. Finally, if I could go back in time to give my younger self some advice, I would tell her not to dwell on what other people may be thinking and not to sweat the small stuff. Instead, it’s more productive to take time to listen to yourself and recognize when you feel like trying something else, leaning in a new direction, or simply taking a break! Well, that’s my story… so far. I’m always learning every day, so all of this is (and will always be) a work in progress. I invite you to share this story along with yours! Write your own article. And to hear more stories about women and minorities working in tech, check out the documentary . Tech Like Us CODE: Debugging the Gender Gap

The New Normal — Engineer, Product Manager, Woman.

Who wants to get a nice and shiny infrastructure diagram?
We are!
Who wants to maintain it?
… Intro That’s right. You work at a company with a lot of infrastructure to manage. There are many services, complex networking, multi-cloud setups, and several CI/CD tools. You have a lot of code written. Maybe with comments. Perhaps even some documentation. Eventually, you start creating diagrams to show your infrastructure at a glance. Everybody loves diagrams! But as soon as you press ‘Save’ button the clock starts ticking, showing how outdated your diagram is. Unfortunately, no one loves outdated stuff. And no one has time to update it. You spent a lot of time being an infrastructure-as-a-code gitops hero. Everything is already here, in the code! There are plenty of tools that transform code into UML diagrams. Is there anything for infrastructure, please? And there is! Let’s take a closer look What’s in the review To make it more practical we are going to use a real repo. The repo has a minimalistic Dockerfile for a simple Spring Boot app. A pinch of Terraform, and a GitHub Actions workflow to glue everything together. It builds the app with Gradle, creates a Docker image, and deploys it to AWS Elastic Container Service. Tools under review Terraform graph
Inframap
Rover
NoReDraw
Others Terraform graph First, let's start with Terraform graph. Terraform provides an embedded option for making a diagram. It creates a Graphviz .dot file. To see a diagram as an image you have to install Graphviz. terraform graph | dot -Tpng > /tmp/graph.png This gives you next diagram Looks acceptable for an out-of-the-box solution. It’s hardly readable, has no coloring scheme and has no icons. I’m not sure if it provides a better sense of architecture than the actual code. Yet, it could be a nice addition to a pull request analysis by demonstrating the actual plan (specify -plan=plan.out). Inframap If the Terraform graph is too basic for your needs, let's move on to Inframap. This tool reads your tfstate or HCL to generate a graph specific to each provider. Instead of showing everything, the tool shows the most important resources. Developers of Inframap created a human-readable version of terraform graph. inframap generate . | dot -Tpng > inframap.png This gives the next diagram Well, that’s way more readable. If you want more app-centric representation that’s the way to go. It doesn't have too much detail (subnets, security groups, IAM roles, etc) and looks more concise. We have all main components like the load balancer, ECS service, and the cluster. A few nitpicks: I would put client instead of im_out. And make the ECS cluster a container (rectangle) with the service inside. Anyway, it’s pretty decent. Rover Next up is Rover, another Terraform visualizer. There are several options for how to run Rover. Unfortunately, I wasn’t able to make Rover run terraform under the hood. Image generation with the specified plan didn’t work for me either. So I had to run Rover in standalone mode and explicitly specify previously generated TF plan: terraform plan -out=plan.out
terraform show -json plan.out > plan.json

docker run --rm -it -p 9000:9000 -v $(pwd):/src im2nguyen/rover:latest -planJSONPath=/src/plan.json The UI on http://localhost:9000/ appeared to be pretty nice. It might be helpful to click around to get more details on what’s going on. Again, mostly for PRs and a general understanding of resources. This is how generated diagram looks like I'm pleased with the color legend and grouping. The information seems to be excessive for the needs of an architecture diagram. Again, it’s a terraform visualizer, it shows everything defined in the plan, with no surprises. Alright, you’ve got the idea of terraform visualizers. They try to represent the state of resources defined in .tf files. Is there anything else left overboard? Right, the applications. The whole point of infrastructure is to serve a set of applications. Without showing the apps it all doesn’t make a lot of sense. Inframap did a great job with its app-centric approach. Are Terraform files not enough for the big picture? Let’s fall down the rabbit hole. NoReDraw This tool takes a step further. Instead of dealing with Terraform, NoReDraw focuses on… everything. Well, currently Gradle, Docker, GitHub Action, and Terraform AWS resources are supported. And ‘supported’ is a bit of a strong word as the project is still in beta. NoReDraw takes a source folder or git repo URL as an input. Then parses and cross-matches everything it can. Want to add more? Implement your own provider as the tool is highly customizable. Don’t like how components are linked together? Create your own LinkStrategy. Another cool benefit is that you can generate a diagram in Plant UML or Mermaid. This gives you an option to embed the drawing directly into your docs as text. wget https://github.com/uladzimirFilipchanka/noredraw/raw/main/noredraw.sh && chmod +x noredraw.sh

./noredraw.sh \
--source . \
--export PNG \
--output noredraw.png Let’s see, from bottom to top. Gradle builds a Jar named arch-diagram-sample. Then it gets containerized by Docker and ends up being a Docker image. The image is assigned to the task which is deployed into Amazon ECS via a GitHub Actions workflow. All correct! The diagram definitely lacks in style, but if you plan to embed it you’ll have to suffer enjoy it anyway. At least some extra coloring wouldn’t hurt. The attributes in blocks nicely describe each artifact and are very handy. I really hope NoReDraw gets more traction in the near future. More providers and features can make it production-ready. Others Finally, let's look at some other tools and services. They didn’t make it to the shortlist but are worth mentioning. Pluralith This service helps to visualize and manage cloud infrastructure by parsing Terraform code. Pluralith goes beyond 'regular' visualizers and offers real-time updates. With nicely done CI/CD integrations (GitHub and GitLab as of now). Lucidscale The older sibling of Lucidchart. It’s a super-powerful tool for managing cloud infrastructure. Lucidscale offers automated and detailed visualizations but works another way. It connects to your cloud account and generates diagrams of the architecture. Currently, AWS, Azure, and GCP are supported. In this case, you don’t even need to adhere to infrastructure-as-a-code. Lucidscale provides all that you may or may not need. Including live sync, filtering, customization, and collaboration. Cloudcraft Like Lucidscale it connects to your cloud account. It offers deep integration with AWS (and now Azure), but focuses on simplicity.
Cloudcraft has a minimalistic and user-friendly interface. It allows you to quickly create cloud architecture diagrams. You might miss some minor features in Cloudcraft, but the beauty of diagrams compensates for it. Outro Yes, there are definitely more tools available on the market (let me know about your favorite ones in the comments). Maintaining up-to-date infrastructure diagrams can be a challenge. But with the right tools, it becomes much more manageable. The key is to find the one that best fits your needs and workflow. Who wants to get a nice and shiny infrastructure diagram? We are! Who wants to maintain it? … Who wants to get a nice and shiny infrastructure diagram? We are! Who wants to maintain it? … Intro That’s right. You work at a company with a lot of infrastructure to manage. There are many services, complex networking, multi-cloud setups, and several CI/CD tools. You have a lot of code written. Maybe with comments. Perhaps even some documentation. Eventually, you start creating diagrams to show your infrastructure at a glance. Everybody loves diagrams! But as soon as you press ‘Save’ button the clock starts ticking, showing how outdated your diagram is. Unfortunately, no one loves outdated stuff. And no one has time to update it. You spent a lot of time being an infrastructure-as-a-code gitops hero. Everything is already here, in the code! There are plenty of tools that transform code into UML diagrams. Is there anything for infrastructure, please? infrastructure-as-a-code And there is! Let’s take a closer look What’s in the review To make it more practical we are going to use a real repo. The repo has a minimalistic Dockerfile for a simple Spring Boot app. A pinch of Terraform, and a GitHub Actions workflow to glue everything together. It builds the app with Gradle, creates a Docker image, and deploys it to AWS Elastic Container Service. Tools under review Terraform graph Inframap Rover NoReDraw Others Terraform graph Terraform graph Inframap Inframap Rover Rover NoReDraw NoReDraw Others Terraform graph First, let's start with Terraform graph . Terraform provides an embedded option for making a diagram. It creates a Graphviz .dot file. To see a diagram as an image you have to install Graphviz . Terraform graph Graphviz terraform graph | dot -Tpng > /tmp/graph.png terraform graph | dot -Tpng > /tmp/graph.png This gives you next diagram Looks acceptable for an out-of-the-box solution. It’s hardly readable, has no coloring scheme and has no icons. I’m not sure if it provides a better sense of architecture than the actual code. Yet, it could be a nice addition to a pull request analysis by demonstrating the actual plan (specify -plan=plan.out ). -plan=plan.out Inframap If the Terraform graph is too basic for your needs, let's move on to Inframap . This tool reads your tfstate or HCL to generate a graph specific to each provider. Instead of showing everything, the tool shows the most important resources. Developers of Inframap created a human-readable version of terraform graph . Inframap terraform graph inframap generate . | dot -Tpng > inframap.png inframap generate . | dot -Tpng > inframap.png This gives the next diagram Well, that’s way more readable. If you want more app-centric representation that’s the way to go. It doesn't have too much detail (subnets, security groups, IAM roles, etc) and looks more concise. We have all main components like the load balancer, ECS service, and the cluster. A few nitpicks: I would put client instead of im_out . And make the ECS cluster a container (rectangle) with the service inside. Anyway, it’s pretty decent. client im_out Rover Next up is Rover , another Terraform visualizer. There are several options for how to run Rover. Unfortunately, I wasn’t able to make Rover run terraform under the hood. Image generation with the specified plan didn’t work for me either. So I had to run Rover in standalone mode and explicitly specify previously generated TF plan: Rover terraform plan -out=plan.out
terraform show -json plan.out > plan.json

docker run --rm -it -p 9000:9000 -v $(pwd):/src im2nguyen/rover:latest -planJSONPath=/src/plan.json terraform plan -out=plan.out
terraform show -json plan.out > plan.json

docker run --rm -it -p 9000:9000 -v $(pwd):/src im2nguyen/rover:latest -planJSONPath=/src/plan.json The UI on http://localhost:9000/ appeared to be pretty nice. It might be helpful to click around to get more details on what’s going on. Again, mostly for PRs and a general understanding of resources. http://localhost:9000/ This is how generated diagram looks like I'm pleased with the color legend and grouping. The information seems to be excessive for the needs of an architecture diagram. Again, it’s a terraform visualizer, it shows everything defined in the plan, with no surprises. Alright, you’ve got the idea of terraform visualizers. They try to represent the state of resources defined in .tf files. Is there anything else left overboard? Right, the applications. The whole point of infrastructure is to serve a set of applications. Without showing the apps it all doesn’t make a lot of sense. Inframap did a great job with its app-centric approach. Are Terraform files not enough for the big picture? Let’s fall down the rabbit hole. NoReDraw This tool takes a step further. Instead of dealing with Terraform, NoReDraw focuses on… everything. Well, currently Gradle, Docker, GitHub Action, and Terraform AWS resources are supported. And ‘supported’ is a bit of a strong word as the project is still in beta. NoReDraw NoReDraw takes a source folder or git repo URL as an input. Then parses and cross-matches everything it can. Want to add more? Implement your own provider as the tool is highly customizable. Don’t like how components are linked together? Create your own LinkStrategy . Another cool benefit is that you can generate a diagram in Plant UML or Mermaid . This gives you an option to embed the drawing directly into your docs as text. LinkStrategy Plant UML Mermaid wget https://github.com/uladzimirFilipchanka/noredraw/raw/main/noredraw.sh && chmod +x noredraw.sh

./noredraw.sh \
--source . \
--export PNG \
--output noredraw.png wget https://github.com/uladzimirFilipchanka/noredraw/raw/main/noredraw.sh && chmod +x noredraw.sh

./noredraw.sh \
--source . \
--export PNG \
--output noredraw.png Let’s see, from bottom to top. Gradle builds a Jar named arch-diagram-sample . Then it gets containerized by Docker and ends up being a Docker image. The image is assigned to the task which is deployed into Amazon ECS via a GitHub Actions workflow. All correct! arch-diagram-sample The diagram definitely lacks in style, but if you plan to embed it you’ll have to suffer enjoy it anyway. At least some extra coloring wouldn’t hurt. The attributes in blocks nicely describe each artifact and are very handy. I really hope NoReDraw gets more traction in the near future. More providers and features can make it production-ready. suffer Others Finally, let's look at some other tools and services. They didn’t make it to the shortlist but are worth mentioning. Pluralith Pluralith This service helps to visualize and manage cloud infrastructure by parsing Terraform code. Pluralith goes beyond 'regular' visualizers and offers real-time updates. With nicely done CI/CD integrations (GitHub and GitLab as of now). Lucidscale Lucidscale The older sibling of Lucidchart. It’s a super-powerful tool for managing cloud infrastructure. Lucidscale offers automated and detailed visualizations but works another way. It connects to your cloud account and generates diagrams of the architecture. Currently, AWS, Azure, and GCP are supported. In this case, you don’t even need to adhere to infrastructure-as-a-code. Lucidscale provides all that you may or may not need. Including live sync, filtering, customization, and collaboration. Cloudcraft Cloudcraft Like Lucidscale it connects to your cloud account. It offers deep integration with AWS (and now Azure), but focuses on simplicity.
Cloudcraft has a minimalistic and user-friendly interface. It allows you to quickly create cloud architecture diagrams. You might miss some minor features in Cloudcraft, but the beauty of diagrams compensates for it. Outro Outro Yes, there are definitely more tools available on the market (let me know about your favorite ones in the comments). Maintaining up-to-date infrastructure diagrams can be a challenge. But with the right tools, it becomes much more manageable. The key is to find the one that best fits your needs and workflow.

Say Goodbye to Outdated Diagrams: Automate Your Infrastructure Visualization

Evolution of Data Sharding Towards Automation and Flexibility in Apache Doris

অ্যাপাচি ডরিসে অটোমেশন এবং নমনীয়তার দিকে ডেটা ভাগ করার বিবর্তন

Entwicklung des Data Sharding hin zu Automatisierung und Flexibilität in Apache Doris

Evolución de la fragmentación de datos hacia la automatización y la flexibilidad en Apache Doris

Evolution du partage de données vers l'automatisation et la flexibilité dans Apache Doris

अपाचे डोरिस में स्वचालन और लचीलेपन की ओर डेटा शार्डिंग का विकास

Apache Doris における自動化と柔軟性に向けたデータ シャーディングの進化

Apache Doris의 자동화 및 유연성을 향한 데이터 샤딩의 진화

Evolução da fragmentação de dados em direção à automação e flexibilidade no Apache Doris

Эволюция сегментирования данных в сторону автоматизации и гибкости в Apache Doris

Apache Doris'te Veri Parçalamanın Otomasyona ve Esnekliğe Doğru Evrimi

Sự phát triển của việc phân chia dữ liệu theo hướng tự động hóa và linh hoạt trong Apache Doris

Apache Doris 中数据分片向自动化和灵活性的演变

Authors:
(1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Table of Links Abstract and 1. Introduction
Computational Approaches in Editorial Design
System
Experimentation and Discussion
Conclusion
Acknowledgments and References Abstract This paper proposes a generative approach for the automatic typesetting of books in desktop publishing. The presented system consists in a computer script that operates inside a widely used design software tool and implements a generative process based on several typographic rules, styles and principles which have been identified in the literature. The performance of the proposed system is tested through an experiment which included the evaluation of its outputs with people. The results reveal the ability of the system to consistently create varied book designs from the same input content as well as visually coherent book designs with different contents while complying with fundamental typographic principles. Keywords Design tools; Data-driven Design; Generative Design; Graphic design; Typography. 1. Introduction Typography is the art of giving our language a visual form. Thus, it is through typography that we materialise and store our knowledge and information [1]. Since the publication of the first typographic book, in the mid-fifteenth century, society has been looking for the most appropriate way to convey a message typographically. In the field of editorial design, this effort has focused on the search for the best principles for designing typographic compositions, such as books [2]. Advances in digital technologies have been changing the work process of graphic designers. The emergence of computational approaches in the design domain enabled designers to explore new perspectives, new conceptual and visual possibilities, and achieve new types of solutions. Furthermore, the emergence of computational approaches comes with a paradigm shift in the role of the designer, who begins to create processes that enable the creation of designs, instead of designing the final solution. In other words, the design concept is translated into a computer program that systematically explores various design possibilities from the original concept. That said, in the particular case of layout design, we consider that the potential of computational approaches is not yet being fully explored. In this work, we explore a computational generative approach for the automatic design of book layouts. The result is a computer system, which operates inside the Adobe InDesign environment, that automatically generates book designs from input content. Figure 1 shows different books created with the presented system. It starts by receiving the input content, namely text and images. Before generating compositions for the input content, the designer can specify restrictions on some of the visual characteristics of the output compositions, e.g. format, size, grid and font. Then, the system automatically typesets the content based on a set of typography rules and principles found in fundamental literature in the field. In the end, the system presents the generated composition to the user as an editable Adobe InDesign document. Overall, the system is capable of creating layout compositions that comply with specific fundamental typographic principles while matching the graphic preferences of the user. In addition, experiments conducted with the system demonstrate its ability to autonomously generate varied compositions with the same input content and also generate visually coherent compositions for different input contents. In addition, it creates functional layout designs in an almost unpredictable manner. This reveals the great potential of this approach to layout design, both in generating outputs that the designer uses as final solutions and in using the outputs as starting points for further explorations. This work is aligned with the experiments previously developed by Ferreira et al. [3] The remaining of the article is organised as follows: Section 2 overviews related works; Section 3 describes the system, namely the interaction process, the engine that runs the system, as well as its inputs and outputs; Section 4 presents an experiment conducted to validate the designs created with the system and discusses the results obtained; Lastly, Section 5 summarises the main contributions of this work and identifies future work. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Authors: Authors: (1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Table of Links Abstract and 1. Introduction Computational Approaches in Editorial Design System Experimentation and Discussion Conclusion Acknowledgments and References Abstract and 1. Introduction Abstract and 1. Introduction Computational Approaches in Editorial Design Computational Approaches in Editorial Design System System Experimentation and Discussion Experimentation and Discussion Conclusion Conclusion Acknowledgments and References Acknowledgments and References Abstract This paper proposes a generative approach for the automatic typesetting of books in desktop publishing. The presented system consists in a computer script that operates inside a widely used design software tool and implements a generative process based on several typographic rules, styles and principles which have been identified in the literature. The performance of the proposed system is tested through an experiment which included the evaluation of its outputs with people. The results reveal the ability of the system to consistently create varied book designs from the same input content as well as visually coherent book designs with different contents while complying with fundamental typographic principles. Keywords Keywords Design tools; Data-driven Design; Generative Design; Graphic design; Typography. 1. Introduction Typography is the art of giving our language a visual form. Thus, it is through typography that we materialise and store our knowledge and information [1]. Since the publication of the first typographic book, in the mid-fifteenth century, society has been looking for the most appropriate way to convey a message typographically. In the field of editorial design, this effort has focused on the search for the best principles for designing typographic compositions, such as books [2]. Advances in digital technologies have been changing the work process of graphic designers. The emergence of computational approaches in the design domain enabled designers to explore new perspectives, new conceptual and visual possibilities, and achieve new types of solutions. Furthermore, the emergence of computational approaches comes with a paradigm shift in the role of the designer, who begins to create processes that enable the creation of designs, instead of designing the final solution. In other words, the design concept is translated into a computer program that systematically explores various design possibilities from the original concept. That said, in the particular case of layout design, we consider that the potential of computational approaches is not yet being fully explored. In this work, we explore a computational generative approach for the automatic design of book layouts. The result is a computer system, which operates inside the Adobe InDesign environment, that automatically generates book designs from input content. Figure 1 shows different books created with the presented system. It starts by receiving the input content, namely text and images. Before generating compositions for the input content, the designer can specify restrictions on some of the visual characteristics of the output compositions, e.g. format, size, grid and font. Then, the system automatically typesets the content based on a set of typography rules and principles found in fundamental literature in the field. In the end, the system presents the generated composition to the user as an editable Adobe InDesign document. Overall, the system is capable of creating layout compositions that comply with specific fundamental typographic principles while matching the graphic preferences of the user. In addition, experiments conducted with the system demonstrate its ability to autonomously generate varied compositions with the same input content and also generate visually coherent compositions for different input contents. In addition, it creates functional layout designs in an almost unpredictable manner. This reveals the great potential of this approach to layout design, both in generating outputs that the designer uses as final solutions and in using the outputs as starting points for further explorations. This work is aligned with the experiments previously developed by Ferreira et al. [3] The remaining of the article is organised as follows: Section 2 overviews related works; Section 3 describes the system, namely the interaction process, the engine that runs the system, as well as its inputs and outputs; Section 4 presents an experiment conducted to validate the designs created with the system and discusses the results obtained; Lastly, Section 5 summarises the main contributions of this work and identifies future work. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Towards the Automation of Book Typesetting: Abstract and Introduction

Authors:
(1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Table of Links Abstract and 1. Introduction
Computational Approaches in Editorial Design
System
Experimentation and Discussion
Conclusion
Acknowledgments and References 5. Conclusion We have presented a novel approach to computationally design books. The presented system implements a generative design process which takes advantage of the scripting capabilities of Adobe InDesign to procedurally typeset books from content provided by the user. We have shown the ability of the system to (i) create book designs that consistently comply with a series of typographic rules, styles and principles identified in the literature; (ii) produce visually diversified books from the same input content; and (iii) produce visually coherent books with different contents. The work presented in the paper may challenge the typical roles of both the tool and the designer. First, by automatically creating and suggesting design alternatives, the tool ends up playing a more active role in the design process. Then, by modifying and developing custom tools, the designer is no longer a mere tool user and becomes the author of tools tailored to specific needs. We believe this shift can be fruitful since it enables the exploration and discovery of new technical and creative possibilities. This work can hopefully provide directions to further research on generative processes for supporting design exploration and finding unique designs. In the particular case of typography, generative approaches such as the one presented in the paper can be useful and reveal great potential, especially in the current print-on-demand market and digital publishing, where each publication may be unique. Our future work will move in the direction of employing Artificial Intelligence techniques, such as Evolutionary Computation and Machine Learning, to enable a deeper exploration of the vast space of book designs that can be achieved with the system and also to automatically suggest settings to designers according to their needs or goals. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Authors: Authors: (1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Table of Links Abstract and 1. Introduction Computational Approaches in Editorial Design System Experimentation and Discussion Conclusion Acknowledgments and References Abstract and 1. Introduction Abstract and 1. Introduction Computational Approaches in Editorial Design Computational Approaches in Editorial Design System System Experimentation and Discussion Experimentation and Discussion Conclusion Conclusion Acknowledgments and References Acknowledgments and References 5. Conclusion We have presented a novel approach to computationally design books. The presented system implements a generative design process which takes advantage of the scripting capabilities of Adobe InDesign to procedurally typeset books from content provided by the user. We have shown the ability of the system to (i) create book designs that consistently comply with a series of typographic rules, styles and principles identified in the literature; (ii) produce visually diversified books from the same input content; and (iii) produce visually coherent books with different contents. The work presented in the paper may challenge the typical roles of both the tool and the designer. First, by automatically creating and suggesting design alternatives, the tool ends up playing a more active role in the design process. Then, by modifying and developing custom tools, the designer is no longer a mere tool user and becomes the author of tools tailored to specific needs. We believe this shift can be fruitful since it enables the exploration and discovery of new technical and creative possibilities. This work can hopefully provide directions to further research on generative processes for supporting design exploration and finding unique designs. In the particular case of typography, generative approaches such as the one presented in the paper can be useful and reveal great potential, especially in the current print-on-demand market and digital publishing, where each publication may be unique. Our future work will move in the direction of employing Artificial Intelligence techniques, such as Evolutionary Computation and Machine Learning, to enable a deeper exploration of the vast space of book designs that can be achieved with the system and also to automatically suggest settings to designers according to their needs or goals. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Towards the Automation of Book Typesetting: Conclusion

Towards the Automation of Book Typesetting: Acknowledgments and References

Authors:
(1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author;
(4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Table of Links Abstract and 1. Introduction
Computational Approaches in Editorial Design
System
Experimentation and Discussion
Conclusion
Acknowledgments and References 2. Computational Approaches in Editorial Design Systematic approaches have been popular in layout design since the mid-twentieth century when some creative practitioners designed layouts based on grids and the variation of the visual features of typographic elements [4–6]. Some works have explored algorithmic approaches for book layout design. In the 1960s, Gerstner introduced a selective and combinatorial method for the design of graphics, including layouts [7]. Afterwards, he translated it into a logical language that computers could understand in Compendium for Literates: A system of Writing [8]. LeWitt compiled, in 1971, a set of formal instructions to design a conceptual art exhibition catalogue [9]. Already in the 1980s, Knuth and Plass [10] presented a dynamic programming algorithm to page breaks avoiding widows and orphans and employed it to typeset a two-column dictionary. Soon thereafter, Knuth introduced the parametric typeface design language, Metafont [11] and the TeX typesetting system [12], enabling anybody to produce a book using a structural markup language and a set of high-level commands. Cooper and her students, at the Visible Language Workshop, experimented with the generation of layouts, e.g. publications that resulted from the collaboration with IBM [13] and the cover for Design Quarterly 142 [14]. In the mid1990s, Maeda also designed a series of digital booklets, the Reactive Book series, where the graphics are controlled interactively by user input [15]. Nevertheless, in the last two decades, we observed the increasing employment of the use of computational graphic design approaches, especially because of the development of easier-to-use creative code environments [16,17]. A solid overview of the field is presented, for instance, by Reas et al. [6] or Richardson [18]. These approaches have been explored in several artistic and creative domains, including the generation of visual and communication artefacts such as poster designs (e.g. Rebelo et al. [19] or Guo et al. [20]), banners (e.g. Gatarski [21] or Yin et al. [22]), user interfaces (e.g. Quiroz et al. [23] or Amitani et al. [24]), visual identities (e.g. Levin [25] or Neue [26]), type designs (e.g. Ahn & Jin [27] or Martins et al. [28]), among others. In the context of book typesetting, two frameworks are stimulating the adoption of computational practices. The library Basil.js [29] provides friendly and accessible tools for scripting and automation in Adobe InDesign. Then, the opensource framework The Magic Book [30] facilitates the design, production, and self-publishing of books. Tailor-made procedural and template-based approaches are employed in the generation and definition of layouts, modification of layouts based on their contents and/or inter-relationships between the elements. LettError type design studio used random processes and parametric design methods to design several typographic artefacts such as calendars, type specimens and even their portfolio [31,32]. Oliveira [33] presented a recursive division method to place elements on both one and multiple-column grid layouts. Cleveland [34] proposed a method for generating style-based design layouts that explores the inter-relationships between text and graphics. Also, he presented a system to generate layouts employing these principles. LUST developed a set of scripts to layout and stylise the book “I Read Where I am” informed by its content [35]. Damera-Venkata et al. [36] presented a template-based probabilistic framework for generating document layouts for variable content. Ahmadullin and Damera-Venkata [37] also presented a probabilistic model for newspaper typeset that, based on given content, divides the available layout into regions and optimises the content to fit within these regions. Flipboard developed Duplo [38], a layout engine that creates news magazines adapted to its contents and based on a set of heuristics such as the amount and flux of text or the existence and position of images. We can also observe the use of Artificial Intelligence approaches for typesetting. Evolutionary computation and greedy approaches have been used to create layouts with varied purposes. For example, Geigel and Loui [39] evolved layouts for photo books by evaluating different aesthetic criteria. Goldenberg [40] employed an evolutionary approach to automatically generate page layouts, minimising the waste of space on the page. Gonzalvez et al. [41] used a greedy simulated annealing algorithm to create multi-column newspaper layouts. Purvis et al. [42] automatically evolved documents using a multi-objective optimisation approach, considering a set of layout constraints and aesthetic measures. Quiroz et al. [43] evolved brochure documents according to user preferences and design guidelines. Strecker and Hennig [44] proposed a grid-based method for newspaper layouts, minimising the wasted space and bearing in mind newspaper design aesthetic measures. Boll et al. [45] and Sandhaus et al. [46,47] evolved photo layouts based on rules of layout design and proposed a method to transform a blog into a photo book considering different aesthetic requirements. Önduygu [48] developed the system Gráphagos that generates compositions through the interactive evolution of specific features of visual elements. Klein [49] developed the tool Crossing, Mixing, Mutating to create variations in a template using genetic operators. Later, an updated version of this tool was released as an InDesign plug-in named Evolving Layout [50]. Lopes et al. [51] developed the system evoDesigner, which automatically creates and evolves designs in the InDesign environment. Recently, Machine Learning approaches have been employed in the layout design field taking into account the relation between elements on layout and the learning of specific typesetting and design styles, e.g. Zheng et al. [52], Li et al. [53], or Kikuchi et al. [54]. Our analysis of the related work unveils a series of computational approaches that reveal great potential for the support and automatisation of the creation of book layouts. However, as far as we know, the existing approaches on this context rarely enable the subsequent modification of the generated designs, even less in the natural working environment of the designer. This way, it is difficult to introduce such computational technologies in the design process in an easy and fluid fashion. Also, we denoted that most of these approaches do not present a multipurpose objective, being developed to generate specific designs. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Authors: Authors: (1) Sérgio M. Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (2) Tiago Martins, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (3) Diogo Ferreira, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal and a Corresponding author; (4) Artur Rebelo, University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal. Table of Links Abstract and 1. Introduction Computational Approaches in Editorial Design System Experimentation and Discussion Conclusion Acknowledgments and References Abstract and 1. Introduction Abstract and 1. Introduction Computational Approaches in Editorial Design Computational Approaches in Editorial Design System System Experimentation and Discussion Experimentation and Discussion Conclusion Conclusion Acknowledgments and References Acknowledgments and References 2. Computational Approaches in Editorial Design Systematic approaches have been popular in layout design since the mid-twentieth century when some creative practitioners designed layouts based on grids and the variation of the visual features of typographic elements [4–6]. Some works have explored algorithmic approaches for book layout design. In the 1960s, Gerstner introduced a selective and combinatorial method for the design of graphics, including layouts [7]. Afterwards, he translated it into a logical language that computers could understand in Compendium for Literates: A system of Writing [8]. LeWitt compiled, in 1971, a set of formal instructions to design a conceptual art exhibition catalogue [9]. Already in the 1980s, Knuth and Plass [10] presented a dynamic programming algorithm to page breaks avoiding widows and orphans and employed it to typeset a two-column dictionary. Soon thereafter, Knuth introduced the parametric typeface design language, Metafont [11] and the TeX typesetting system [12], enabling anybody to produce a book using a structural markup language and a set of high-level commands. Cooper and her students, at the Visible Language Workshop, experimented with the generation of layouts, e.g. publications that resulted from the collaboration with IBM [13] and the cover for Design Quarterly 142 [14]. In the mid1990s, Maeda also designed a series of digital booklets, the Reactive Book series, where the graphics are controlled interactively by user input [15]. Nevertheless, in the last two decades, we observed the increasing employment of the use of computational graphic design approaches, especially because of the development of easier-to-use creative code environments [16,17]. A solid overview of the field is presented, for instance, by Reas et al. [6] or Richardson [18]. These approaches have been explored in several artistic and creative domains, including the generation of visual and communication artefacts such as poster designs (e.g. Rebelo et al. [19] or Guo et al. [20]), banners (e.g. Gatarski [21] or Yin et al. [22]), user interfaces (e.g. Quiroz et al. [23] or Amitani et al. [24]), visual identities (e.g. Levin [25] or Neue [26]), type designs (e.g. Ahn & Jin [27] or Martins et al. [28]), among others. In the context of book typesetting, two frameworks are stimulating the adoption of computational practices. The library Basil.js [29] provides friendly and accessible tools for scripting and automation in Adobe InDesign. Then, the opensource framework The Magic Book [30] facilitates the design, production, and self-publishing of books. Tailor-made procedural and template-based approaches are employed in the generation and definition of layouts, modification of layouts based on their contents and/or inter-relationships between the elements. LettError type design studio used random processes and parametric design methods to design several typographic artefacts such as calendars, type specimens and even their portfolio [31,32]. Oliveira [33] presented a recursive division method to place elements on both one and multiple-column grid layouts. Cleveland [34] proposed a method for generating style-based design layouts that explores the inter-relationships between text and graphics. Also, he presented a system to generate layouts employing these principles. LUST developed a set of scripts to layout and stylise the book “I Read Where I am” informed by its content [35]. Damera-Venkata et al. [36] presented a template-based probabilistic framework for generating document layouts for variable content. Ahmadullin and Damera-Venkata [37] also presented a probabilistic model for newspaper typeset that, based on given content, divides the available layout into regions and optimises the content to fit within these regions. Flipboard developed Duplo [38], a layout engine that creates news magazines adapted to its contents and based on a set of heuristics such as the amount and flux of text or the existence and position of images. We can also observe the use of Artificial Intelligence approaches for typesetting. Evolutionary computation and greedy approaches have been used to create layouts with varied purposes. For example, Geigel and Loui [39] evolved layouts for photo books by evaluating different aesthetic criteria. Goldenberg [40] employed an evolutionary approach to automatically generate page layouts, minimising the waste of space on the page. Gonzalvez et al. [41] used a greedy simulated annealing algorithm to create multi-column newspaper layouts. Purvis et al. [42] automatically evolved documents using a multi-objective optimisation approach, considering a set of layout constraints and aesthetic measures. Quiroz et al. [43] evolved brochure documents according to user preferences and design guidelines. Strecker and Hennig [44] proposed a grid-based method for newspaper layouts, minimising the wasted space and bearing in mind newspaper design aesthetic measures. Boll et al. [45] and Sandhaus et al. [46,47] evolved photo layouts based on rules of layout design and proposed a method to transform a blog into a photo book considering different aesthetic requirements. Önduygu [48] developed the system Gráphagos that generates compositions through the interactive evolution of specific features of visual elements. Klein [49] developed the tool Crossing, Mixing, Mutating to create variations in a template using genetic operators. Later, an updated version of this tool was released as an InDesign plug-in named Evolving Layout [50]. Lopes et al. [51] developed the system evoDesigner, which automatically creates and evolves designs in the InDesign environment. Recently, Machine Learning approaches have been employed in the layout design field taking into account the relation between elements on layout and the learning of specific typesetting and design styles, e.g. Zheng et al. [52], Li et al. [53], or Kikuchi et al. [54]. Our analysis of the related work unveils a series of computational approaches that reveal great potential for the support and automatisation of the creation of book layouts. However, as far as we know, the existing approaches on this context rarely enable the subsequent modification of the generated designs, even less in the natural working environment of the designer. This way, it is difficult to introduce such computational technologies in the design process in an easy and fluid fashion. Also, we denoted that most of these approaches do not present a multipurpose objective, being developed to generate specific designs. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Towards the Automation of Book Typesetting: Computational Approaches in Editorial Design

Towards the Automation of Book Typesetting: System

Towards the Automation of Book Typesetting: Experimentation and Discussion

Self-driving cars always seem to be just out of reach. Technology improves and people make bold claims about when they will become the norm, only for new setbacks to further delay their inevitable rollout. Autonomous long-haul trucks, however, could arrive earlier. Semis often have relatively straightforward routes. Driverless AI would have an easier time navigating more predictable highway traffic than pedestrian-heavy inner cities. Still, the future of autonomous big rigs in the supply chain is uncertain. Businesses and government bodies should consider the potential consequences — both positive and negative. Positive Consequences of Autonomous Trucks Supply chains would gain a lot from driverless trucking. When this technology becomes reliable enough for logistics professionals to use, it could transform the industry for the better. 1. Improved Safety The most important advantage of autonomous trucks is that once they become accurate enough, they’re safer. As common as driving is, it’s a dangerous profession. That’s largely because humans make a lot of risky mistakes. A whopping 60% of people admit to driving sleepy, slowing their reaction time and making them likely to miss incoming hazards. Drunk and distracted drivers pose even more risks. Robots can’t get tired or intoxicated and don’t text, so they’d eliminate these dangers from the supply chain. Self-driving features can improve driver safety even before fully autonomous semis are a thing. Automated lane assist and braking can fill the gaps when truckers have momentary lapses in judgment. 2. Faster Shipping Driverless vehicles would also make supply chains more efficient. While autonomous trucks won’t necessarily drive faster than humans, they can travel for longer. Truckers can only drive for 11 hours and be on-duty — driving or not — for 14 hours before taking a 10-hour break. This law is an important part of road safety, but it significantly extends shipping times. However, it wouldn’t apply to an autonomous truck. Self-driving vehicles could travel indefinitely, only stopping for fuel. As a result, shipments would reach their destinations in a fraction of the time. Such speed would mean lower operating costs and less disruption for companies, and greater convenience for consumers. 3. Overcoming Worker Shortages Similarly, autonomous trucks would help logistics businesses become more productive. A driver shortage has emerged as e-commerce has boomed, with fewer people entering the trucking sector. Some industry insiders say the nation is short 64,000 truck drivers as of 2023, and almost one-third are 55 or older. Before long, those aging employees will retire, worsening the labor gap. Self-driving vehicles are a convenient solution to this problem. A smaller workforce isn’t a big issue when businesses don’t need as many workers to reach the same level of productivity. Overcoming the trucker shortage lets companies run shipments more frequently. Consequently, delivery timelines would fall even further, and supply chains could prevent disruptions. 4. More Sustainability Driverless semis also bring environmental benefits. It’s no secret that transportation needs to improve its sustainability. The sector consumes 125 million gallons of diesel every day, leading to huge greenhouse gas emissions. While autonomous tech wouldn’t eliminate those numbers, it can reduce them. Trucks waste a lot of fuel through poor driving practices like idling and harsh acceleration. Businesses can program a self-driving big rig to avoid these behaviors, minimizing the diesel they consume. Emissions will fall as supply chains use less fuel to do the same work. Similar programming steps could also stretch electric vehicle ranges. That way, the industry could switch to carbon-free shipping methods without sacrificing productivity. Negative Consequences of Autonomous Trucks As impressive as those benefits are, autonomous trucks also have their fair share of negative implications. The downsides deserve just as much attention for self-driving technology to proceed safely. 1. Safety Concerns Driverless semis will be safer than human truckers once they reach a certain reliability standard. The problem is that they could be more dangerous until they get to that point. Self-driving AI has to anticipate billions of potential situations, and even then, it can only interpret things if its sensors work well. Right now, things like rain, snow, fog, and even reflections pose difficulties, leading to crashes. People can also misread signs and fail to see incoming obstacles, but they’re at least more familiar with common hazards than driverless cars. Traffic involves a lot of unpredictable behavior and subtle nuances people take for granted. AI struggles to understand such complexities, which could lead to devastating consequences in the right situation. 2. Regulatory Complications Autonomous trucks also raise legal questions when events like this happen. Currently, there are no specific laws for determining liability with self-driving cars. Who’s responsible for an accident involving one of these semis? The logistics company? The truck manufacturer? It’s harder to decide without another human party. Insuring a driverless truck may be challenging without concrete regulations on the topic. Consequently, supply chain organizations will hesitate to embrace them before laws catch up with technology. Legislation will eventually adapt, but it will slow adoption before it does. 3. Job Loss As with any type of automation, there are questions about fair employment practices. The transportation industry may face a labor shortage today, but it still employs 3.54 million drivers in the U.S. Self-driving vehicles may threaten these people’s jobs. One would hope that companies would use autonomous semis to fill the gap instead of taking all open positions, but who’s to say they will? The temptation to cut labor costs may lead businesses to automate all shipments. History proves technology creates more jobs than it destroys in the long run. However, these new roles require a different skill set. Consequently, while employment may even out over time, driverless technology could cause some initial job disruption. Self-Driving Trucks Face an Uncertain Future It’s still unclear when autonomous trucking will become a viable option. When it does, businesses have several complications to work through. Driverless vehicles could be a massive help for companies and the customers they serve. At the same time, improper use or regulation of this technology could worsen things more than it improves. Organizations and government agencies should consider these questions today to ensure a clearer, safer future. Self-driving cars always seem to be just out of reach. Technology improves and people make bold claims about when they will become the norm, only for new setbacks to further delay their inevitable rollout. Autonomous long-haul trucks, however, could arrive earlier. Self-driving cars always seem to be just out of reach. Technology improves and people make bold claims about when they will become the norm, only for new setbacks to further delay their inevitable rollout. Autonomous long-haul trucks, however, could arrive earlier. Semis often have relatively straightforward routes. Driverless AI would have an easier time navigating more predictable highway traffic than pedestrian-heavy inner cities. Still, the future of autonomous big rigs in the supply chain is uncertain. Businesses and government bodies should consider the potential consequences — both positive and negative. Positive Consequences of Autonomous Trucks Supply chains would gain a lot from driverless trucking. When this technology becomes reliable enough for logistics professionals to use, it could transform the industry for the better. 1. Improved Safety The most important advantage of autonomous trucks is that once they become accurate enough, they’re safer. As common as driving is, it’s a dangerous profession. That’s largely because humans make a lot of risky mistakes. A whopping 60% of people admit to driving sleepy , slowing their reaction time and making them likely to miss incoming hazards. Drunk and distracted drivers pose even more risks. Robots can’t get tired or intoxicated and don’t text, so they’d eliminate these dangers from the supply chain. 60% of people admit to driving sleepy 60% of people admit to driving sleepy Self-driving features can improve driver safety even before fully autonomous semis are a thing. Automated lane assist and braking can fill the gaps when truckers have momentary lapses in judgment. 2. Faster Shipping Driverless vehicles would also make supply chains more efficient. While autonomous trucks won’t necessarily drive faster than humans, they can travel for longer. Truckers can only drive for 11 hours and be on-duty — driving or not — for 14 hours before taking a 10-hour break. This law is an important part of road safety, but it significantly extends shipping times. However, it wouldn’t apply to an autonomous truck. only drive for 11 hours only drive for 11 hours Self-driving vehicles could travel indefinitely, only stopping for fuel. As a result, shipments would reach their destinations in a fraction of the time. Such speed would mean lower operating costs and less disruption for companies, and greater convenience for consumers. 3. Overcoming Worker Shortages Similarly, autonomous trucks would help logistics businesses become more productive. A driver shortage has emerged as e-commerce has boomed, with fewer people entering the trucking sector. Some industry insiders say the nation is short 64,000 truck drivers as of 2023, and almost one-third are 55 or older. short 64,000 truck drivers short 64,000 truck drivers Before long, those aging employees will retire, worsening the labor gap. Self-driving vehicles are a convenient solution to this problem. A smaller workforce isn’t a big issue when businesses don’t need as many workers to reach the same level of productivity. Overcoming the trucker shortage lets companies run shipments more frequently. Consequently, delivery timelines would fall even further, and supply chains could prevent disruptions. 4. More Sustainability Driverless semis also bring environmental benefits. It’s no secret that transportation needs to improve its sustainability. The sector consumes 125 million gallons of diesel every day, leading to huge greenhouse gas emissions. While autonomous tech wouldn’t eliminate those numbers, it can reduce them. 125 million gallons of diesel 125 million gallons of diesel Trucks waste a lot of fuel through poor driving practices like idling and harsh acceleration. Businesses can program a self-driving big rig to avoid these behaviors, minimizing the diesel they consume. Emissions will fall as supply chains use less fuel to do the same work. Similar programming steps could also stretch electric vehicle ranges. That way, the industry could switch to carbon-free shipping methods without sacrificing productivity. Negative Consequences of Autonomous Trucks As impressive as those benefits are, autonomous trucks also have their fair share of negative implications. The downsides deserve just as much attention for self-driving technology to proceed safely. 1. Safety Concerns Driverless semis will be safer than human truckers once they reach a certain reliability standard. The problem is that they could be more dangerous until they get to that point. Self-driving AI has to anticipate billions of potential situations , and even then, it can only interpret things if its sensors work well. Right now, things like rain, snow, fog, and even reflections pose difficulties, leading to crashes. People can also misread signs and fail to see incoming obstacles, but they’re at least more familiar with common hazards than driverless cars. billions of potential situations billions of potential situations Traffic involves a lot of unpredictable behavior and subtle nuances people take for granted. AI struggles to understand such complexities, which could lead to devastating consequences in the right situation. 2. Regulatory Complications Autonomous trucks also raise legal questions when events like this happen. Currently, there are no specific laws for determining liability with self-driving cars. Who’s responsible for an accident involving one of these semis? The logistics company? The truck manufacturer? It’s harder to decide without another human party. no specific laws for determining liability no specific laws for determining liability Insuring a driverless truck may be challenging without concrete regulations on the topic. Consequently, supply chain organizations will hesitate to embrace them before laws catch up with technology. Legislation will eventually adapt, but it will slow adoption before it does. 3. Job Loss As with any type of automation, there are questions about fair employment practices. The transportation industry may face a labor shortage today, but it still employs 3.54 million drivers in the U.S. Self-driving vehicles may threaten these people’s jobs. employs 3.54 million drivers employs 3.54 million drivers One would hope that companies would use autonomous semis to fill the gap instead of taking all open positions, but who’s to say they will? The temptation to cut labor costs may lead businesses to automate all shipments. History proves technology creates more jobs than it destroys in the long run. However, these new roles require a different skill set. Consequently, while employment may even out over time, driverless technology could cause some initial job disruption. Self-Driving Trucks Face an Uncertain Future It’s still unclear when autonomous trucking will become a viable option. When it does, businesses have several complications to work through. Driverless vehicles could be a massive help for companies and the customers they serve. At the same time, improper use or regulation of this technology could worsen things more than it improves. Organizations and government agencies should consider these questions today to ensure a clearer, safer future.

7 Consequences of Autonomous Trucks in the Supply Chain

Nowadays, automation technologies are an indispensable part of our daily lives—the impact of AI, ML, and robotic process automation (RPA) in business is clear with labor productivity in a sector such as manufacturing set to increase to 37% by 2025, according to Deloitte. Innovations in automation are also allowing us to be more streamlined and efficient with employees free from the burden of mundane tasks, but the prospect of going fully automated is a completely different story altogether. Let’s dive into what the factors are that might be holding companies back from advancing automation processes, how businesses can identify areas that need to be automated, and what the best practices are for developing a strategic plan for successful automation. What Is Holding Companies Back from Fully Advancing Automation Processes? Despite the obvious benefits, several factors in play prevent companies from fully biting the automation bullet. Unfortunately, everyone’s wallets are tight at the moment and high initial costs and budget constraints are not easy to get over, particularly for small businesses.Implementing advanced automation technologies often requires substantial upfront investment in software, hardware, and training. The latter is a pressing issue, given that81% of IT professionals think that they can use AI, but only 12% have the skills to do so. And 70% of workers likely need to upgrade their AI skills. Automation technologies demand a workforce skilled in AI, ML, and data analytics, but there is a widespread skills gap, with many employees lacking the necessary expertise to implement and manage these technologies effectively. On top of that, even after identifying the automation to go with, integration can be really difficult. Trying to incorporate new systems in something that might be out-of-date can lead to compatibility issues and, in turn, security issues that might crop up as a result of sensitive data being leaked. The final issue to mention is one that is harder to address than any other—resistance to change within the organization, which though it may seem something trivial, can have a huge impact on the ability to transition. Many people in management positions will continue to see automation as a threat and unnecessary distraction. How to Identify Where Automation Can Take Effect One of the crucial first steps that companies can take is with process mapping and analysis. This allows you to be abreast of existing workflows and identify tasks that are clearly repetitive, time-consuming tasks that can be automated, such as some data entry, payroll and customer service. Process mining and AI-driven process discovery can uncover some valuable insights into these workflows. And don’t forget to speak to frontline workers, as they often know exactly what inefficiencies exist in their daily tasks and can suggest areas where automation could have a significant impact.It’s also a good idea to use the concept of a pilot program whereby you test out AI in a specific department. This involves several crucial steps such as optimizing the model parameters of the AI, deciding on the length of time you are going to take with the pilot, and picking employees who are keen to take on the challenge. For example, say there is a consumer goods company that wants to implement automation in their supply chain. In order to do this, they would have to identify repetitive tasks and start a pilot program in a low-risk area of their operation. Something like inventory management would be perfect in this case, as it can have very clear rules surrounding stock levels. The Best Practices for a Strategic Automation Plan Companies need to first define some goals for what they aim to achieve with automation, this might be reducing operational costs, improving productivity, or even enhancing customer service. Whichever priority it is, you need to include all the relevant stakeholders, including IT and the aforementioned workers who are on the frontline. Investing in training and development is another critical component. You need to look both internally and externally for this, there is plenty of official documentation from platforms such as TensorFlow or PyTorch that offer tutorials and examples you can use. Additionally, open-source AI projects can allow employees to learn from the community and get real-life practical experience.Ultimately choosing the right form of automation is not just a numbers game, you need to consider your team’s capabilities and the ease of use. You should also think about what upgrades might be necessary for your tech stack , how easy the product is to maintain and what compatibility there is with existing technologies so as to prevent security issues mentioned earlier. Lastly, you need to ensure that the data you are using is valid to prevent inaccurate results that might require you to redo automation tests. Final Thoughts Transitioning to an entirely automated future is a complex process and one that is not fool-proof by any means. However, it is on the horizon and approaching fast so we have to face it sooner or later—Gartner predicts that 70% of organizations are set to apply automation in processes by 2025. Additionally, given that 30% of expenses have already been reduced as a result of automation technology in business, the trend is only going to continue. Companies that look to embrace automation and develop strategic plans for its implementation will be more successful. They will be able to maintain and grow their customer base while simply completing more straightforward tasks in a faster time frame. The future of automation is promising, but companies need to make sure that they allocate their resources wisely and train their staff properly to make the transition one that doesn’t disrupt operations. Shagun Malhotra, Founder of SkyStem LLC Nowadays, automation technologies are an indispensable part of our daily lives—the impact of AI, ML , and robotic process automation (RPA) in business is clear with labor productivity in a sector such as manufacturing set to increase to 37% by 2025 , according to Deloitte. Innovations in automation are also allowing us to be more streamlined and efficient with employees free from the burden of mundane tasks, but the prospect of going fully automated is a completely different story altogether. AI, ML set to increase to 37% by 2025 Let’s dive into what the factors are that might be holding companies back from advancing automation processes, how businesses can identify areas that need to be automated, and what the best practices are for developing a strategic plan for successful automation. What Is Holding Companies Back from Fully Advancing Automation Processes? Despite the obvious benefits, several factors in play prevent companies from fully biting the automation bullet. Unfortunately, everyone’s wallets are tight at the moment and high initial costs and budget constraints are not easy to get over, particularly for small businesses. Implementing advanced automation technologies often requires substantial upfront investment in software, hardware, and training. The latter is a pressing issue, given that 81% of IT professionals think that they can use AI , but only 12% have the skills to do so. And 70% of workers likely need to upgrade their AI skills. 81% of IT professionals think that they can use AI Automation technologies demand a workforce skilled in AI, ML, and data analytics, but there is a widespread skills gap, with many employees lacking the necessary expertise to implement and manage these technologies effectively. On top of that, even after identifying the automation to go with, integration can be really difficult. Trying to incorporate new systems in something that might be out-of-date can lead to compatibility issues and, in turn, security issues that might crop up as a result of sensitive data being leaked. The final issue to mention is one that is harder to address than any other—resistance to change within the organization, which though it may seem something trivial, can have a huge impact on the ability to transition. Many people in management positions will continue to see automation as a threat and unnecessary distraction. How to Identify Where Automation Can Take Effect One of the crucial first steps that companies can take is with process mapping and analysis. This allows you to be abreast of existing workflows and identify tasks that are clearly repetitive, time-consuming tasks that can be automated, such as some data entry, payroll and customer service. Process mining and AI-driven process discovery can uncover some valuable insights into these workflows. And don’t forget to speak to frontline workers, as they often know exactly what inefficiencies exist in their daily tasks and can suggest areas where automation could have a significant impact. It’s also a good idea to use the concept of a pilot program whereby you test out AI in a specific department. This involves several crucial steps such as optimizing the model parameters of the AI, deciding on the length of time you are going to take with the pilot, and picking employees who are keen to take on the challenge. For example, say there is a consumer goods company that wants to implement automation in their supply chain. In order to do this, they would have to identify repetitive tasks and start a pilot program in a low-risk area of their operation. Something like inventory management would be perfect in this case, as it can have very clear rules surrounding stock levels. The Best Practices for a Strategic Automation Plan Companies need to first define some goals for what they aim to achieve with automation, this might be reducing operational costs, improving productivity, or even enhancing customer service. Whichever priority it is, you need to include all the relevant stakeholders, including IT and the aforementioned workers who are on the frontline. Investing in training and development is another critical component. You need to look both internally and externally for this, there is plenty of official documentation from platforms such as TensorFlow or PyTorch that offer tutorials and examples you can use. Additionally, open-source AI projects can allow employees to learn from the community and get real-life practical experience. Ultimately choosing the right form of automation is not just a numbers game, you need to consider your team’s capabilities and the ease of use. You should also think about what upgrades might be necessary for your tech stack , how easy the product is to maintain and what compatibility there is with existing technologies so as to prevent security issues mentioned earlier. Lastly, you need to ensure that the data you are using is valid to prevent inaccurate results that might require you to redo automation tests. Final Thoughts Transitioning to an entirely automated future is a complex process and one that is not fool-proof by any means. However, it is on the horizon and approaching fast so we have to face it sooner or later— Gartner predicts that 70% of organizations are set to apply automation in processes by 2025. Additionally, given that 30% of expenses have already been reduced as a result of automation technology in business, the trend is only going to continue. Gartner predicts that 70% of organizations Companies that look to embrace automation and develop strategic plans for its implementation will be more successful. They will be able to maintain and grow their customer base while simply completing more straightforward tasks in a faster time frame. The future of automation is promising, but companies need to make sure that they allocate their resources wisely and train their staff properly to make the transition one that doesn’t disrupt operations. Shagun Malhotra, Founder of SkyStem LLC Shagun Malhotra, Founder of SkyStem LLC Shagun Malhotra, Founder of SkyStem LLC Shagun Malhotra , Founder of SkyStem LLC Shagun Malhotra SkyStem LLC

The Trade-Offs Companies Face with Automation

AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort: Experiments

Authors:
(1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn);
(2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn);
(3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn);
(4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn);
(5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com);
(6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction
Related Work
Our Method
Experiments
Conclusion and References
Appendix 3 OUR METHOD The goal of our method is to generate diverse storytelling images of high quality and with minimal human effort. Considering the complexity of scenes in storytelling images, our general idea is to combine the comprehension and planning capabilities of LLMs [et al 2023; OpenAI 2023] and the generation ability of the large-scale text-to-image models [Rombach et al. 2022]. The pipeline is shown in Fig. 2, which can be divided into a condition preparation stage in (a) and a conditional image generation stage in (b). Specifically, we first utilize LLMs to convert the textual descriptions of stories into layouts of the storytelling images, as detailed in Sec. 3.1. To improve the quality of generated story images, we propose a simple yet effective method to transform sparse bounding boxes into dense control signals like sketches or keypoints, without introducing manual labor (detailed in Sec. 3.2). Subsequently, we generate story images with a reasonable scene arrangement based on the layout, as detailed in Sec. 3.3. Finally, we propose a method to eliminate the requirement for users to collect training data for each character, enabling the generation of identity-consistent story images from only texts (detailed in Sec. 3.4). Since our approach only fine-tunes the pre-trained text-to-image image diffusion model on a few images, we can easily leverage existing models on civiati[1] for storytelling in arbitrary characters, scenes, and even styles. 3.1 Story to Layout Generation Layout Generation. After dividing the story into panel descriptions, we leverage LLMs to extract the scene layout from each panel description, as shown in the following equation: 3.2 Dense Condition Generation Motivation. Although using sparse bounding boxes as a control signal can improve the generation of subjects and obtain more reasonable scene layouts, we find that it cannot consistently produce high-quality generation results. There are cases where the images do not exactly match the scene layout or the generated images are of low quality, as detailed in the experiments in Sec. 4.4. We believe that this is mainly due to the limited information provided by the bounding boxes. The model faces difficulties in generating a large amount of content all at once, with limited guidance. For this reason, we propose to improve the final story image generation by introducing dense sketch or keypoint guidances. To this end, we devise a dense condition generation module based on the layout generated in the previous section, as shown in Fig. 2(d). Note that the process of composing dense conditions is fully automatic and does not require any manual interaction. 3.3 Controllable Storytelling Image Generation Large-scale pre-trained text-to-image models, such as Stable Diffusion [Rombach et al. 2022], are capable of generating images from text. However, limited by the language comprehension ability of the text encoder in the model, and the incorrect association between text and image regions in the generation process, the directly generated images often suffer from a series of problems such as object missing, attribution confusion, etc [Chefer et al. 2023]. To tackle this, we introduce additional control signals to improve the quality of image generation. 3.4 Eliminating Character-wise Data Collection Identity Consistency. We propose a training-free consistency modeling method to meet the requirement of identity consistency, as shown in Fig. 3 (d). Specifically, we treat multiple images of a single character as different frames in a video and generate them simultaneously using a pre-trained diffusion model. In this process, the self-attention in the generative model is expanded to other “video frames" [Wang et al. 2023; Wu et al. 2022] to strengthen the dependencies among images, thus obtaining identity-consistent generation results. Concretely, in self-attention, we let the latent features in each frame attend to the features in the first and previous frames to build the dependency. The process can be represented as: Subsequently, we extract sketches for non-human characters and keypoints for human characters from these images. Finally, we use T2I-Adapter to inject the control guidance into the latent feature of corresponding frames in the generation process. In addition, in order to further ensure the quality of the generated data, we use CLIP score to filter the generated data, and select the images that are consistent with the text descriptions as the training data for customized generation. Discussion. In this section, we combine the proposed training-free identity-consistency modeling method with the viewpoint conditioned image translation model to achieve both identity consistency and diversity in character generation. A simpler approach is to directly use the multi-view images from the view-point conditioned image translation model as training data for customization. However, we found that the directly generated results often suffer from distortions or large differences in the color and texture of the images from different viewpoints (see Sec. 4.4 for details). For this reason, we need to leverage the above consistency modeling approach to obtain both texture- and structure-consistent images for each character. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. [1] https://civitai.com Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Authors: Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction Related Work Our Method Experiments Conclusion and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Our Method Our Method Experiments Experiments Conclusion and References Conclusion and References Appendix Appendix 3 OUR METHOD The goal of our method is to generate diverse storytelling images of high quality and with minimal human effort. Considering the complexity of scenes in storytelling images, our general idea is to combine the comprehension and planning capabilities of LLMs [et al 2023; OpenAI 2023] and the generation ability of the large-scale text-to-image models [Rombach et al. 2022]. The pipeline is shown in Fig. 2, which can be divided into a condition preparation stage in (a) and a conditional image generation stage in (b). Specifically, we first utilize LLMs to convert the textual descriptions of stories into layouts of the storytelling images, as detailed in Sec. 3.1. To improve the quality of generated story images, we propose a simple yet effective method to transform sparse bounding boxes into dense control signals like sketches or keypoints, without introducing manual labor (detailed in Sec. 3.2). Subsequently, we generate story images with a reasonable scene arrangement based on the layout, as detailed in Sec. 3.3. Finally, we propose a method to eliminate the requirement for users to collect training data for each character, enabling the generation of identity-consistent story images from only texts (detailed in Sec. 3.4). Since our approach only fine-tunes the pre-trained text-to-image image diffusion model on a few images, we can easily leverage existing models on civiati[1] for storytelling in arbitrary characters, scenes, and even styles. 3.1 Story to Layout Generation Layout Generation. After dividing the story into panel descriptions, we leverage LLMs to extract the scene layout from each panel description, as shown in the following equation: Layout Generation. 3.2 Dense Condition Generation Motivation . Although using sparse bounding boxes as a control signal can improve the generation of subjects and obtain more reasonable scene layouts, we find that it cannot consistently produce high-quality generation results. There are cases where the images do not exactly match the scene layout or the generated images are of low quality, as detailed in the experiments in Sec. 4.4. Motivation We believe that this is mainly due to the limited information provided by the bounding boxes. The model faces difficulties in generating a large amount of content all at once, with limited guidance. For this reason, we propose to improve the final story image generation by introducing dense sketch or keypoint guidances. To this end, we devise a dense condition generation module based on the layout generated in the previous section, as shown in Fig. 2(d). Note that the process of composing dense conditions is fully automatic and does not require any manual interaction. 3.3 Controllable Storytelling Image Generation Large-scale pre-trained text-to-image models, such as Stable Diffusion [Rombach et al. 2022], are capable of generating images from text. However, limited by the language comprehension ability of the text encoder in the model, and the incorrect association between text and image regions in the generation process, the directly generated images often suffer from a series of problems such as object missing, attribution confusion, etc [Chefer et al. 2023]. To tackle this, we introduce additional control signals to improve the quality of image generation. 3.4 Eliminating Character-wise Data Collection Identity Consistency. We propose a training-free consistency modeling method to meet the requirement of identity consistency, as shown in Fig. 3 (d). Specifically, we treat multiple images of a single character as different frames in a video and generate them simultaneously using a pre-trained diffusion model. In this process, the self-attention in the generative model is expanded to other “video frames" [Wang et al. 2023; Wu et al. 2022] to strengthen the dependencies among images, thus obtaining identity-consistent generation results. Concretely, in self-attention, we let the latent features in each frame attend to the features in the first and previous frames to build the dependency. The process can be represented as: Identity Consistency. Subsequently, we extract sketches for non-human characters and keypoints for human characters from these images. Finally, we use T2I-Adapter to inject the control guidance into the latent feature of corresponding frames in the generation process. In addition, in order to further ensure the quality of the generated data, we use CLIP score to filter the generated data, and select the images that are consistent with the text descriptions as the training data for customized generation. Discussion . In this section, we combine the proposed training-free identity-consistency modeling method with the viewpoint conditioned image translation model to achieve both identity consistency and diversity in character generation. A simpler approach is to directly use the multi-view images from the view-point conditioned image translation model as training data for customization. However, we found that the directly generated results often suffer from distortions or large differences in the color and texture of the images from different viewpoints (see Sec. 4.4 for details). For this reason, we need to leverage the above consistency modeling approach to obtain both texture- and structure-consistent images for each character. Discussion This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv [1] https://civitai.com

AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort: Our Method

Authors:
(1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn);
(2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn);
(3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn);
(4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn);
(5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com);
(6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction
Related Work
Our Method
Experiments
Conclusion and References
Appendix 2 RELATED WORK 2.1 Story Visualization Story visualization aims to generate a series of visually consistent images from a story described in text. Limited by the generative capacity of the model, many story visualization approaches [Chen et al. 2022; Li 2022; Li et al. 2019; Maharana and Bansal 2021; Maharana et al. 2021, 2022; Pan et al. 2022; Rahman et al. 2022; Song et al. 2020] seek to largely simplify the task such that it becomes tractable, by considering specific characters, scenes, and image styles in a particular dataset. Early story visualization methods are mostly built upon GANs [Goodfellow et al. 2020]. For example, StoryGAN [Li et al. 2019] pioneers the story visualization task by proposing a GAN-based framework that considers both the full story and the current sentence for coherent image generation. CP-CSV [Song et al. 2020], DuCo-StoryGAN [Maharana et al. 2021], and VLC-StoryGAN [Li et al. 2019] follow the GAN-based framework, while improving the consistency of storytelling via better character-preserving or text understanding. Difference from these works, VP-CSV [Chen et al. 2022] leverages VQ-VAE a transformer-based language model for story visualization. StoryDALL-E [Maharana et al. 2022] leverages the pre-trained DALL-E [Ramesh et al. 2021] for better story visualization and proposes a novel task named story continuation that supports story visualization with a given initial image. AR-LDM [Pan et al. 2022] proposes a diffusion model-based method that generates story images in an autoregressive manner. While progress has been made, these methods rely on storyspecific training on datasets like PororoSV [Li et al. 2019] and FlintstonesSV [Maharana and Bansal 2021], making it difficult to generalize these methods to varying characters and scenes. The development of large-scale pre-trained text-to-image generative models [Ramesh et al. 2022, 2021; Rombach et al. 2022; Saharia et al. 2022] opens up new opportunities for generalizable story visualization. Several attempts have been made to generate storytelling images with diverse characters [Gong et al. 2023; Jeong et al. 2023; Liu et al. 2023c]. Jeong et al. [Jeong et al. 2023] utilized textual inversion [Gal et al. 2022] to swap human identities in story images, thus generalizing the characters in story visualization. However, the identity is not well preserved, and the method is limited to a single human character in storytelling. Intelligent Grimm [Liu et al. 2023c] proposes the task of open-ended visual storytelling. They collect a dataset of children’s storybooks and train an autoregressive generative model for story visualization. The limitation is clear: they focus on the storytelling of the children’s storybook style, and it needs to re-train the model to generalize to other styles, contents, etc., which is not scalable. Probably the most similar work to ours is TaleCraft [Gong et al. 2023], which also proposes a systematic pipeline for story visualization. Note that, they require user-provided sketches for each character in each story image to obtain visually pleasing generations, which can be laborious to obtain. Moreover, all existing methods rely on multiple user-provided images for each character to obtain identity-coherent story visualizations. In contrast, our method allows for generating diverse and coherent story visualization results with only text descriptions as inputs. 2.2 Controllable Image Generation The scaling of text-image paired data [Schuhmann et al. 2022], computational resources, and model size have enabled unprecedented text-to-image (T2I) generation results [Ramesh et al. 2022, 2021; Rombach et al. 2022; Saharia et al. 2022]. Large-scale pre-trained text-to-image models, such as Stable Diffusion [Rombach et al. 2022], are capable of generating images from text, i.e., 𝐼 = DM (𝑝), where DM(·) is the pre-trained diffusion model and 𝑝 is the text prompt that describe the image 𝐼. In this process, the text information is passed into the image’s latent representation through cross-attention layers in the model. The attention [Vaswani et al. 2017] operation can be written as However, limited by the language understanding capability of the text encoder and poor text-to-image content association [Chefer et al. 2023], T2I models, like Stable Diffusion [Rombach et al. 2022], can perform poorly in the generation of multiple characters and complex scenes [Chefer et al. 2023]. To alleviate this drawback, some approaches introduce explicit spatial guidance in T2I generative models. For example, ControlNet [Zhang and Agrawala 2023] uses zero convolution layers and a trainable copy of the original model weights, introducing reliable control in diffusion models. T2I-Adapter [Mou et al. 2023] achieves control ability by proposing the adapter that extracts guidance feature and adds it to the feature from the corresponding UNet encoder. GLIGEN [Li et al. 2023] injects a gated self-attention block into the UNet, enabling the model to make good use of the grounding inputs. Inspired by the ability of large language models (LLMs) [et al 2023; OpenAI 2023] being able to understand and plan, recent works [Feng et al. 2023; Lian et al. 2023] employ LLM for layout generation. Specifically, LayoutGPT [Feng et al. 2023] achieves plausible results in 2D image layouts and even 3D indoor scene synthesis by applying in-context learning on LLMs. LLM-grounded Diffusion [Lian et al. 2023] proposes a two-stage process based on the LLM-generated layout and local prompts. Specifically, it first generates the local objects within each bounding box based on the corresponding local prompt, and then re-generates the final result based on the inversed latent of local objects. While effective, LLMgrounded Diffusion requires careful hyper-parameter tuning for the trade-off between structural guidance and inter-object relationship modeling. Moreover, it is difficult for the users to control the detailed structure of the generated objects. In contrast, we use the intuitive sketch or keypoint to guide the final image generation. Thus, we can not only achieve high-quality story image generation, but also allow interactive story visualization by simply tuning the generated sketch or keypoint conditions. 2.3 Customized Image Generation Story visualization requires that the identities of characters and scenes in a story remain consistent across different images. Customized image generation can meet this requirement to a large extent. Early methods [Gal et al. 2022; Ruiz et al. 2022] focus on the customized generation of a single object. For example, DreamBooth [Ruiz et al. 2022] fine tunes the pre-trained T2I diffusion model under a class-specific prior-preservation loss. Textual Inversion [Gal et al. 2022] enables customized generation by inverting subject image content into text embeddings. Unlike these approaches, Custom Diffusion [Kumari et al. 2022] further achieves multi-subject customization by combining the multiple customization weights through closed-form constrained optimization. Cones [Liu et al. 2023a] finds that a small cluster of concept neurons in the diffusion model corresponds to a single subject, and thus achieves customized generation of multiple objects by combining these concept neurons. Cones2 [Liu et al. 2023f] further achieves more effective multi-object customization by combining text embedding of different concepts with simple layout control. Differently, Mix-of-Show [Gu et al. 2023] proposes gradient fusion to effectively combine multiple customized LoRA [Hu et al. 2022] weights and performs multi-object customization with the aid of the T2I-Adapter’s dense controls. While significant progress has been made, existing methods perform poorly on one-shot customization. The training data for subject-driven generation has to be identity-consistent and diverse. As a result, existing story visualization methods require multiple user-provided images for each character. To tackle this issue, we propose a training-free consistency modeling method, and leverage the 3D prior in 3D-aware generative models [Liu et al. 2023d,b] to obtain multi-view consistent character images for customized generation, thus eliminating the reliance on human labor to collect or draw character images. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Authors: Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction Related Work Our Method Experiments Conclusion and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Our Method Our Method Experiments Experiments Conclusion and References Conclusion and References Appendix Appendix 2 RELATED WORK 2.1 Story Visualization Story visualization aims to generate a series of visually consistent images from a story described in text. Limited by the generative capacity of the model, many story visualization approaches [Chen et al. 2022; Li 2022; Li et al. 2019; Maharana and Bansal 2021; Maharana et al. 2021, 2022; Pan et al. 2022; Rahman et al. 2022; Song et al. 2020] seek to largely simplify the task such that it becomes tractable, by considering specific characters, scenes, and image styles in a particular dataset. Early story visualization methods are mostly built upon GANs [Goodfellow et al. 2020]. For example, StoryGAN [Li et al. 2019] pioneers the story visualization task by proposing a GAN-based framework that considers both the full story and the current sentence for coherent image generation. CP-CSV [Song et al. 2020], DuCo-StoryGAN [Maharana et al. 2021], and VLC-StoryGAN [Li et al. 2019] follow the GAN-based framework, while improving the consistency of storytelling via better character-preserving or text understanding. Difference from these works, VP-CSV [Chen et al. 2022] leverages VQ-VAE a transformer-based language model for story visualization. StoryDALL-E [Maharana et al. 2022] leverages the pre-trained DALL-E [Ramesh et al. 2021] for better story visualization and proposes a novel task named story continuation that supports story visualization with a given initial image. AR-LDM [Pan et al. 2022] proposes a diffusion model-based method that generates story images in an autoregressive manner. While progress has been made, these methods rely on storyspecific training on datasets like PororoSV [Li et al. 2019] and FlintstonesSV [Maharana and Bansal 2021], making it difficult to generalize these methods to varying characters and scenes. The development of large-scale pre-trained text-to-image generative models [Ramesh et al. 2022, 2021; Rombach et al. 2022; Saharia et al. 2022] opens up new opportunities for generalizable story visualization. Several attempts have been made to generate storytelling images with diverse characters [Gong et al. 2023; Jeong et al. 2023; Liu et al. 2023c]. Jeong et al. [Jeong et al. 2023] utilized textual inversion [Gal et al. 2022] to swap human identities in story images, thus generalizing the characters in story visualization. However, the identity is not well preserved, and the method is limited to a single human character in storytelling. Intelligent Grimm [Liu et al. 2023c] proposes the task of open-ended visual storytelling. They collect a dataset of children’s storybooks and train an autoregressive generative model for story visualization. The limitation is clear: they focus on the storytelling of the children’s storybook style, and it needs to re-train the model to generalize to other styles, contents, etc., which is not scalable. Probably the most similar work to ours is TaleCraft [Gong et al. 2023], which also proposes a systematic pipeline for story visualization. Note that, they require user-provided sketches for each character in each story image to obtain visually pleasing generations, which can be laborious to obtain. Moreover, all existing methods rely on multiple user-provided images for each character to obtain identity-coherent story visualizations. In contrast, our method allows for generating diverse and coherent story visualization results with only text descriptions as inputs. 2.2 Controllable Image Generation The scaling of text-image paired data [Schuhmann et al. 2022], computational resources, and model size have enabled unprecedented text-to-image (T2I) generation results [Ramesh et al. 2022, 2021; Rombach et al. 2022; Saharia et al. 2022]. Large-scale pre-trained text-to-image models, such as Stable Diffusion [Rombach et al. 2022], are capable of generating images from text, i.e., 𝐼 = DM (𝑝), where DM(·) is the pre-trained diffusion model and 𝑝 is the text prompt that describe the image 𝐼. In this process, the text information is passed into the image’s latent representation through cross-attention layers in the model. The attention [Vaswani et al. 2017] operation can be written as However, limited by the language understanding capability of the text encoder and poor text-to-image content association [Chefer et al. 2023], T2I models, like Stable Diffusion [Rombach et al. 2022], can perform poorly in the generation of multiple characters and complex scenes [Chefer et al. 2023]. To alleviate this drawback, some approaches introduce explicit spatial guidance in T2I generative models. For example, ControlNet [Zhang and Agrawala 2023] uses zero convolution layers and a trainable copy of the original model weights, introducing reliable control in diffusion models. T2I-Adapter [Mou et al. 2023] achieves control ability by proposing the adapter that extracts guidance feature and adds it to the feature from the corresponding UNet encoder. GLIGEN [Li et al. 2023] injects a gated self-attention block into the UNet, enabling the model to make good use of the grounding inputs. Inspired by the ability of large language models (LLMs) [et al 2023; OpenAI 2023] being able to understand and plan, recent works [Feng et al. 2023; Lian et al. 2023] employ LLM for layout generation. Specifically, LayoutGPT [Feng et al. 2023] achieves plausible results in 2D image layouts and even 3D indoor scene synthesis by applying in-context learning on LLMs. LLM-grounded Diffusion [Lian et al. 2023] proposes a two-stage process based on the LLM-generated layout and local prompts. Specifically, it first generates the local objects within each bounding box based on the corresponding local prompt, and then re-generates the final result based on the inversed latent of local objects. While effective, LLMgrounded Diffusion requires careful hyper-parameter tuning for the trade-off between structural guidance and inter-object relationship modeling. Moreover, it is difficult for the users to control the detailed structure of the generated objects. In contrast, we use the intuitive sketch or keypoint to guide the final image generation. Thus, we can not only achieve high-quality story image generation, but also allow interactive story visualization by simply tuning the generated sketch or keypoint conditions. 2.3 Customized Image Generation Story visualization requires that the identities of characters and scenes in a story remain consistent across different images. Customized image generation can meet this requirement to a large extent. Early methods [Gal et al. 2022; Ruiz et al. 2022] focus on the customized generation of a single object. For example, DreamBooth [Ruiz et al. 2022] fine tunes the pre-trained T2I diffusion model under a class-specific prior-preservation loss. Textual Inversion [Gal et al. 2022] enables customized generation by inverting subject image content into text embeddings. Unlike these approaches, Custom Diffusion [Kumari et al. 2022] further achieves multi-subject customization by combining the multiple customization weights through closed-form constrained optimization. Cones [Liu et al. 2023a] finds that a small cluster of concept neurons in the diffusion model corresponds to a single subject, and thus achieves customized generation of multiple objects by combining these concept neurons. Cones2 [Liu et al. 2023f] further achieves more effective multi-object customization by combining text embedding of different concepts with simple layout control. Differently, Mix-of-Show [Gu et al. 2023] proposes gradient fusion to effectively combine multiple customized LoRA [Hu et al. 2022] weights and performs multi-object customization with the aid of the T2I-Adapter’s dense controls. While significant progress has been made, existing methods perform poorly on one-shot customization. The training data for subject-driven generation has to be identity-consistent and diverse. As a result, existing story visualization methods require multiple user-provided images for each character. To tackle this issue, we propose a training-free consistency modeling method, and leverage the 3D prior in 3D-aware generative models [Liu et al. 2023d,b] to obtain multi-view consistent character images for customized generation, thus eliminating the reliance on human labor to collect or draw character images. one-shot This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort: Related Work

AutoStory: Generating Diverse Storytelling Images with Minimal Effort: Conclusion and References

AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort: Abstract and Intro

Authors:
(1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn);
(2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn);
(3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn);
(4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn);
(5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com);
(6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction
Related Work
Our Method
Experiments
Conclusion and References
Appendix A APPENDIX A.1 More Implementation Details Detailed Prompts for the LLM. As described in Sec. 3 in the main text, we utilize LLMs to accomplish the story and layout generation. Specifically, we leverage the LLM for (1) generating the story, (2) dividing the story into panels, and (3) generating prompts and layout from the panels. In implementation, we further split the third step into two sub-steps, where we first convert the text of each panel into prompts suitable for generating the image, and then parse the prompts into the layout and local prompts. The detailed prompts and sampled LLM outputs are shown in Fig. 11. More Details on the Main Results. Fig. 4 shows the story image generation results of our method with varying characters, storylines, and image styles. Here, we present the character images used to train the customized model for each story. The story visualization results in the left two columns in Fig. 4 are obtained with the user-supplied character images. The corresponding characters are shown in Fig. 9. Differently, the story visualization results in the right two columns are obtained with only the story texts as inputs, and the characters are automatically generated by our method. The generated images for each character are shown in Fig. 10. It can be seen that the animal and human characters generated by our method are of high quality and consistent identities. The images of a single animal character show high diversity, with the orientation of the bird and the cat changing constantly from left to right. The human characters, however, are slightly less diverse, with a lower degree of variance in facial orientation. We believe that this is mainly due to the fact that the diffusion model is trained primarily on humans with frontal faces, making it difficult to generate side-facing images. Nonetheless, the character image data generated by our method can be effectively used for training customization models in story visualization, without introducing overfitting. It is worth mentioning that even though the character Tom in our generated data wears suits, we can generate images with Tom wearing a T-shirt after we specify that the character wears a T-shirt in the local prompt, as shown in the story visualization in Fig. 4 (d). Moreover, the characteristics of Tom are well-maintained, such as the shape of his face and the white hair. This indicates that the customized model trained with our generated data learns the character’s identity without overfitting. A.2 Intermediate Results Visualization To better understand our approach, in this section, we visualize the intermediate process of generating a single story image, as shown in Fig. 12 and Fig. 13. We first generate singlecharacter images based on the Local prompts generated by LLM, as shown in (a) and (b). The perception models, including Grounding-SAM, PidiNet, and HRNet, are then utilized to obtain the keypoints of human characters, or sketches of nonhuman characters, as shown in (c) and (d). Subsequently, the LLM-generated layout is utilized to compose the keypoints or sketches of individual subjects into a dense condition for generating story images, as shown in (d). Finally based on the dense conditions, prompts, and layout, we generate the story image as shown in (f). A.3 More Story Visualization Results In Fig. 14 and Fig. 15, we showcase more story visualization results of our method. As can be seen, our AutoStory can produce high-quality, text-aligned, and identity-consistent story images, even when generating long stories. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Authors: Authors: (1) Wen Wang, Zhejiang University, Hangzhou, China and Equal Contribution (wwenxyz@zju.edu.cn); (2) Canyu Zhao, Zhejiang University, Hangzhou, China and Equal Contribution (volcverse@zju.edu.cn); (3) Hao Chen, Zhejiang University, Hangzhou, China (haochen.cad@zju.edu.cn); (4) Zhekai Chen, Zhejiang University, Hangzhou, China (chenzhekai@zju.edu.cn); (5) Kecheng Zheng, Zhejiang University, Hangzhou, China (zkechengzk@gmail.com); (6) Chunhua Shen, Zhejiang University, Hangzhou, China (chunhuashen@zju.edu.cn). Table of Links Abstract and Introduction Related Work Our Method Experiments Conclusion and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Our Method Our Method Experiments Experiments Conclusion and References Conclusion and References Appendix Appendix A APPENDIX A.1 More Implementation Details Detailed Prompts for the LLM. As described in Sec. 3 in the main text, we utilize LLMs to accomplish the story and layout generation. Specifically, we leverage the LLM for (1) generating the story, (2) dividing the story into panels, and (3) generating prompts and layout from the panels. In implementation, we further split the third step into two sub-steps, where we first convert the text of each panel into prompts suitable for generating the image, and then parse the prompts into the layout and local prompts. The detailed prompts and sampled LLM outputs are shown in Fig. 11. Detailed Prompts for the LLM. More Details on the Main Results. Fig. 4 shows the story image generation results of our method with varying characters, storylines, and image styles. Here, we present the character images used to train the customized model for each story. The story visualization results in the left two columns in Fig. 4 are obtained with the user-supplied character images. The corresponding characters are shown in Fig. 9. Differently, the story visualization results in the right two columns are obtained with only the story texts as inputs, and the characters are automatically generated by our method. The generated images for each character are shown in Fig. 10. It can be seen that the animal and human characters generated by our method are of high quality and consistent identities. The images of a single animal character show high diversity, with the orientation of the bird and the cat changing constantly from left to right. The human characters, however, are slightly less diverse, with a lower degree of variance in facial orientation. We believe that this is mainly due to the fact that the diffusion model is trained primarily on humans with frontal faces, making it difficult to generate side-facing images. Nonetheless, the character image data generated by our method can be effectively used for training customization models in story visualization, without introducing overfitting. It is worth mentioning that even though the character Tom in our generated data wears suits, we can generate images with Tom wearing a T-shirt after we specify that the character wears a T-shirt in the local prompt, as shown in the story visualization in Fig. 4 (d). Moreover, the characteristics of Tom are well-maintained, such as the shape of his face and the white hair. This indicates that the customized model trained with our generated data learns the character’s identity without overfitting. More Details on the Main Results. A.2 Intermediate Results Visualization To better understand our approach, in this section, we visualize the intermediate process of generating a single story image, as shown in Fig. 12 and Fig. 13. We first generate singlecharacter images based on the Local prompts generated by LLM, as shown in (a) and (b). The perception models, including Grounding-SAM, PidiNet, and HRNet, are then utilized to obtain the keypoints of human characters, or sketches of nonhuman characters, as shown in (c) and (d). Subsequently, the LLM-generated layout is utilized to compose the keypoints or sketches of individual subjects into a dense condition for generating story images, as shown in (d). Finally based on the dense conditions, prompts, and layout, we generate the story image as shown in (f). A.3 More Story Visualization Results In Fig. 14 and Fig. 15, we showcase more story visualization results of our method. As can be seen, our AutoStory can produce high-quality, text-aligned, and identity-consistent story images, even when generating long stories. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Stories, News & Mentions about AKTO

Akto’s Stories on HackerNoon