Ubiquitous Computing: Smart Devices, Environments, and Interactions

ubiquitous computing

Stefan Poslad

Ubiquitous computing Intelligent devices, environments and interactions

Stefan Poslad Queen Mary, University of London, UK

A publication of John Wiley and Sons, Ltd.

This edition was first published in 2009 OR 2009 John Wiley & Sons Ltd. Registered Office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK Information on how to apply for permission to reuse material Copyright Protected This book is available on our website at www.wiley.com. The author's right to be identified as the author of this work has been asserted under the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, except as permitted by Copyright Law, UK Designs & Patents 1988. without the prior permission of the publisher. Wiley also publishes his books in various electronic formats. Some print content may not be available in eBooks. The designations used by companies to distinguish their products are often claimed as trademarks. All brand and product names used in this book are trade names, service marks, trademarks, or registered trademarks of their respective holders. The publisher is not affiliated with any of the products or vendors mentioned in this book. This publication is intended to provide accurate and reliable information related to the subject matter. It is sold on the condition that the publisher does not render professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloging in Publication Data Poslad, Stefan. Ubiquitous Computing: Smart Device, Environment, and Interactions / Stefan Poslad. p. cm. Includes index. ISBN 978-0-470-03560-3 (substance) 1. Ubiquitous computing. 2. Context-aware computing. 3. Human-Computer Interaction. I. Title. QA76.5915.P67 2009 004—dc22 2008052234 A catalog entry for this book is available from the British Library. ISBN 978-0-470-03560-3 (H/B) Established in Times of 9/11 Points by Integra Software Services Pvt. Limited Pondicherry, India Printed and bound in Great Britain by Antony Rowe, Chippenham, UK

To my family, Ros and Ben here, and friends and family in three beautiful parts of the world, South Wales (UK), Glandorf and Brisbane.

content

List of figures

xix

table list

XXIII

Preface Acknowledgments 1 Ubiquitous Computing: Fundamentals and Vision 1.1 Living in a Digital World 1.1.1 Chapter Summary 1.1.2 Illustrative Applications of Ubiquitous Computing 1.1.2.1 Personal Reminders 1.1.2.2 Adaptive Scheduled Transportation 1.1.2.3 Grocery Management 1.1.2.4 Supply order 1.1. 3 Holistic Framework for UbiCom: Smart DEI 1.2 Modeling the key properties of ubiquitous computing 1.2.1 Core properties of UbiCom systems 1.2.2 Distributed ICT systems 1.2.2.1 Networked ICT devices 1.2.2.2 Transparency and openness 1.2. 3 Implicit Human-Computer Interaction (iHCI) 1.2.3.1 The Silent Computer 1.2.3.2 Implicit vs. Explicit Human-Computer Interaction 1.2.3.3 Embedded vs. Virtual, Augmented, and Mediated Reality 1.2.4 Context Awareness 1.2.4.1 Three Main Types of environmental context: physical, user, Virtual 1.2.4.2 Knowledge of the user 1.2.4.3 Knowledge of the active context vs. passive 1.2.5 Autonomy 1.2.5.1 Reduction of human interaction Action 1.2.5.2 Ease of maintenance of the system compared to self-retaining systems

xxvii 1 1 2 2 3 5 5 6 7 8 9 9 10 10 11 11 12 12 13 14 14 15 15 16 16

content

viii

1.2.6 Intelligence 1.2.7 Taxonomy of UbiCom properties 1.3 Generalized system-environment interaction 1.3.1 Human-ICT device interaction (HCI) 1.3.2 ICT device-physical world interaction (CPI) 1.4 Architectural design for UbiCom systems: Smart DEI Model 1.4.1 Smart Devices 1.4.1.1 Weiser ICT Device Shapes: Tabs, Pads, and Boards 1.4.1.2 Advanced Shapes for ICT Devices: Dust, Skin, and Sound 1.4.1.3 Mobility 1.4 1.4 Access to Ephemeral Services 1.4.1.5 Localized and Proprietary-Aware 1.4.2 Intelligent Environments 1.4.2.1 Labeling, Sensing, and Control Environments 1.4.2.2 Integrated vs. Unconnected 1.4.2.3 Device Sizes 1.4.3 Intelligent Interaction 1.4.3.1 Basic Interaction 1.4.3.2 Intelligent Interaction 1.5 Discussion 1.5.1 System Interconnection Properties, Environments, and Designs 1.5.2 Common Myths about Ubiquitous Computing 1.5.3 Organization of the Smart DEI Approach Exercises References 2 Application en and requirements 2.1 Introduction 2.1.1 Overview 2.2 Example of early UbiCom research projects 2.2 .1 Smart devices: CCI 2.2.1.1 Smart boards, pads and tabs 2.2.1.2 Active badge, bat and floor 2.2.2 Smart environments : CPI and CCI 2.2.2.1 Classroom 2000 2.2.2.2 Smart space and meeting room 2.2.2.3 Interactive Workspaces and iRoom 2.2.2.4 Cooltown 2.2.2.5 EasyLiving and SPOT 2.2.2.6 HomeLab and Ambient Intelligence 2.2.3 Smart devices: CPI 2.2 .3.1 Unimate and MH-1 Robots 2.2.3.2 Smart Dust and TinyOS 2.2.4 Smart Devices: iHCI and HPI 2.2 .4.1 Calm Computing 2.2.4.2 Things That Think and Tangible Bits 2.2.4.3 DataTiles 2.2.4.4 Ambient Wood 2.2.4.5 WearComp and WearCam 2.2.4.6 Cyborg 1.0 and 2.0 2.2.5 Other UbiCom projects

17 17 22 24 25 26 27 28 28 29 29 30 30 31 31 32 32 32 33 34 34 35 37 38 39 41 41 41 41 42 42 42 43 43 43 44 45 46 46 46 49 5 2 0 4 8 5

content

2.3 Everyday applications in the virtual, human and physical world 2.3.1 Ubiquitous device networks: CCI 2.3.2 Human-computer interaction 2.3.2.1 Widespread access to audio and video content 2.3.2.2 Widespread access to information and e-books 2.3. 2.3 Universal Local control of ICT systems 2.3.2.4 User knowledge and personal spaces 2.3.3 Person-to-person interaction (HHI) applications 2.3.3.1 Transaction-based e-commerce and m-commerce services 2.3.3.2 Increased productivity of mobile people 2.3 . 3.3 Community care 2.3.4 Human-world physical-computer interaction (HPI) and (CPI) 2.3.4.1 Awareness of the physical environment 2.3.4.2 Control of the (physical) environment 2.3.4.3 Smart public services 2.3.4.4 Smart buildings and home automation 2.3.4.5 Smart home environments and smart furniture 2.3.4.6 Smart street furniture 2.3.4.7 Smart vehicles, transportation and travel 2.3.4.8 Ubiquitous games and social physical spaces 2.4 Discussion 2.4.1 Achievements of early projects and current status 2.4 .1.1 Smart De 2.4.1.2 Smart Physical World Environments 2.4.1.3 Context Awareness and Service Discovery 2.4.1.4 Smart Wearable Devices and Implants Exercises References

3 Intelligent Devices and Services 3.1 Introduction 3.1.1 Chapter Overview 3.1.2 Characteristics of Intelligent Devices and Services 3.1.3 Distributed Systems Perspective 3.1.4 Abstraction vs. Virtualization 3.2 Service Architecture Models 3.2.1 Partitioning and service component distribution 3.2.2 Multi-tier client service models 3.2.2.1 Distributed data storage 3.2.2.2 Distributed processing 3.2.2.3 Client-server design 3.2.2.4 Proxy-based service access 3.2.3 Middleware 3.2 .4 Service Oriented Computing (SOC) 3.2.5 Grid Computing 3.2. 6 Peer-to-peer systems 3.2.7 Device models 3.3 Service delivery lifecycle 3.3.1 Network discovery 3.3.2 Service advertisement, discovery, selection and configuration

53 53 54 54 55 56 58 58 59 59 60 61 61 61 62 62 63 65 65 66 67 67 67 68 69 69 71 71

75 75 75 75 77 78 80 80 81 82 82 83 84 85 86 87 88 91 91 92 93

content

3.3.2.1 Web service discovery 3.3.2.2 Semantic Web and semantic resource discovery 3.3.3 Service invocation 3.3.3.1 Distributed processes 3.3.3.2 Asynchronous (MOM) versus synchronous (RPC) communication models 3.3.3.3 Reliable versus unreliable communication 3.3.3.4 Caches 3.3.3.5 On-Demand Service Access 3.3.3.6 Event-Driven Architectures (EDA) 3.3.3.7 Shared Data Repository 3.3.3.8 Enterprise Service Bus (ESB) Model 3.3.3.9 Volatile Service Invocation 3.3 .4 Service composition 3.3.4.1 Service interoperability 3.4 Virtual machines and operating systems 3.4.1 Virtual machines 3.4.2 BIOS 3.4.3 Multitasking operating systems (MTOS) 3.4.4 Process control 3.4.5 Memory management 3.4.6 Entry and Exit Exercises References

95 95 95 96 97 99 99 100 101 103 103 104 105 106 106 106 107 108 109 110 111 111 112

4 Mobile Smart Devices, Cards, and Device Networks 4.1 Introduction 4.1.1 Chapter Overview 4.2 Mobile Smart Devices, Users, Resources, and Code 4.2.1 Mobile Service Design 4.2.1.1 SMS and Mobile Web Services 4.2.1.2 Java VM and J2ME 4.2.1.3. NET CF 4.2.2 Mobile code 4.2.3 Mobile devices and mobile users 4.3 Operating systems for mobile computers and communication devices 4.3.1 Microkernel designs 4.3.2 Mobility support 4.3.3 Devices with limited resources 4.3.4 Power management 4.3.4.1 Low Performance CPU 4.3.4.2 Application Support 4.4 Smart Card Devices 4.4.1 Smart Card Operating System 4.4.2 Smart Card Development 4.5 Device Networking 4.5.1 HAVi, HES, and X10 4.5.2 Detection of devices 4.5.3 OSGi Exercises References

115 115 115 115 116 117 119 120 121 122 123 123 123 124 125 125 126 126 127 128 128 129 129 131 132 133

content

5 Human-Computer Interaction 5.1 Introduction 5.1.1 Chapter Overview 5.1.2 Explicit HCI: Motivation and Properties 5.1.3 Complexity of Generalized Explicit HCI 5.1.4 Implicit HCI: Motivation and Properties 5.2 User Interfaces and Interaction for Four Devices popular 5.2. 1 Variety of interaction with ICT devices 5.2.2 PC interface 5.2.3 Mobile handheld device interfaces 5.2.3.1 Limited key input handling: multi-touch, T9, Fastap, softkeys and virtual keyboard 5.2.3.2 Limited output handling 5.2. 4 Game Console Interfaces and Interaction 5.2.5 Localized Remote Control: Video Devices 5.3 Hidden User Interface Through Simple Smart Devices 5.3.1 Multimodal Visual Interfaces 5.3.2 Gesture Interfaces 5.3.3 Reflective Versus Active Displays 5.3 .4 Combination of user input and output Interfaces 5.3 .4.1 Touch screens 5.3.4.2 Tangible interfaces 5.3.4.3 Organic interfaces 5.3.5 Auditory interfaces 5.3.6 Natural language interfaces 5.4 Hidden UI through portable and implanted devices 5.4.1 Post-human technological model 5.4.2 Virtual reality and augmented reality 5.4.3 Interaction with laptops 5.4.3.1 Head-up display (HUD) 5.4.3.2 Eyetap 5.4.3.3 Virtual retinal display (VRD) 5.4.3.4 Clothing as a computer 5.4. 4 Computer Implants and Brain-Computer Interfaces 5.4.5 Sense of Presence and Telepresence 5.5 Human-Centered Design (HCD) 5.5.1 Human-Centered Design Life Cycle 5.5.2 Methods for Gathering User Information and create used models 5.5. 3 Definition of the context of use of the virtual and physical environment 5.5.4 Definition of the context of use and the requirements of the human environment 5.5.4.1 User characteristics 5.5.5 Interaction design 5.5.5.1 Conceptual models and mental models 5.5.6 Evaluation 5.6 User Models: Capture and Representation 5.6.1 Indirect User Input and Modeling 5.6.2 Direct User Input and Modeling 5.6.3 User Stereotypes 5.6.4 Modeling User Tasks and Planned Goals 5.6.5 Multiple Tasks users and activity-based computing 5.6.6 Situation and action models versus planned action models

135 135 135 136 136 136 137 138 138 139 140 140 141 142 143 143 144 144 145 147 148 148 149 149 150 151 151 152 152 152 153 154 154 154 155 155 157 157 158 158 159 160 160 160 160 161 162 162 163 164 164 165 166 167 167 167

content

XIII

5.7 Design iHCI 5.7.1 Characteristics of the iHCI model 5.7.2 Knowledge of the user context 5.7.3 More intuitive and personalized interaction 5.7.4 Personalization 5.7.5 Affective computing: Interactions using the emotional context of users 5.7.6 Heuristics and patterns Design Exercises References 6 Labeling, Capturing, and Control 6.1 Introduction 6.1.1 Chapter Summary 6.2 Labeling in the Physical World 6.2.1 Life Cycle for Labeling Physical Objects 6.2.2 Labels: Types and Properties 6.2.3 Managing Physical Labels and virtual 6.2.4 RFID tags 6.2 . 4.1 Active RFID Tags 6.2.4.2 Passive RFID Tags 6.2.5 Personalized and Social Tags 6.2.6 Micro and Macro Tags 6.3 Sensors and Sensor Networks 6.3.1 Overview of Sensor Network Components and Processes 6.3.2 System Electronics Sensor 6.3.3 Physical Network: Environment, Density, and Transmission 6.3.4 Data Network: Addressing and Routing 6.3.4.1 Sensor Networks vs. Ad Hoc Networks 6.3.5 Data Processing: Distributed Data Storage and Data Retrieval 6.4 Microactuation and detection: MEMS 6.4.1 Manufacturing 6.4.2 Micro Actuators 6.4.3 Microsensors 6.4.4 Smart surfaces, skin, paint, matter and dust 6.4.5 Reduction to nanotechnology and quantum devices 6.5 Embedded systems and real-time systems 6.5. 1 Application Specific Operating Systems (ASOS) 6.5.2 Real-time operating systems for embedded systems 6.6 Control systems (for physical world tasks) 6.6.1 Programmable controllers 6.6.2 Simple PID controllers 6.6.3 More complex controllers 6.7 Robots 6.7.1 Robotic Manipulators 6.7.2 Mobile Robots 6.7.3 Biologically Inspired Robots 6.7.4 Nanobots 6.7.5 Robotic Application Development UbiCom Exercises References

167 167 168 168 169 171 171 175 175 175 179 179 180 180 181 181 183 183 185 185 185 187 187 187 187 189 191 192 193 194 195 195 197 197 19 19 19 2002 202 203 203 203 204 205 206 207 207 209 219 210 210

content

XIII

7 Context-Aware Systems 7.1 Introduction 7.1.1 Chapter Overview 7.1.2 Context-Aware Applications 7.2 Modeling Context-Aware Systems 7.2.1 Context Types 7.2.2 Context Creation and Composition 7.2.3 Context-Aware Personalization 7.2 .4 Environment modeling 7.2.5 Context representation 7.2.6 A basic architecture 7.2.7 Challenges in context perception 7.3 Mobility perception 7.3.1 Call routing for mobile users 7.3.2 Cell phone location determination 7.3.3 Mobile user perception as an example of composite context: knowledge 7.3.4 Tourism services for mobile users 7.4 Spatial awareness 7.4.1 Spatial context generation 7.4.1.1 Spatial detection 7.4.1.2 Location awareness 7.4.2 Location and others Spatial Abstractions 7.4.3 User Context Generation and Context-Aware Personalization 7.4. 3.1 Cartography: adaptation of spatial points of view to different user contexts 7.4.3.2 Geocoding: assignment of location contexts to user contexts 7.4.4 Query and management of spatial contexts: GIS 7.5 Temporal perception: Coord 7.5.1 Clock synchronization : temporal context creation 7.5.2 Temporal models and abstractions 7.5.3 Temporal context management and adaptation to user contexts 7.6 ICT system knowledge 7.6.1 Context-aware presentation and interaction in the user interface 7.6.1.1 Capture of the user interface context 7.6.1.2 Personalization of content 7.6.2 Personalization of network-enabled service Exercises References

213 213 214 214 216 216 218 219 221 221 222 225 227 227 227

8 Intelligent Systems (IS) By Patricia Charlton 8.1 Introduction 8.1.1 Chapter Summary 8.2 Basic Concepts 8.2.1 Types of Intelligent Systems 8.2.2 Types of Environments for Intelligent Systems 8.2.3 Use of Intelligence in Ubiquitous Computing

228 228 229 230 230 231 233 233 233 234 234 235 235 236 237 238 238 238 239 240 242 242 245 245 246 246 246 247 248

content

xiv

8.3 IS architectures 8.3.1 What a model knows and how to use it 8.3.1.1 Types of architectural models 8.3.1.2 One-sided versus two-sided system environment models 8.3.1.3 Model representations 8.3.1.4 How system models are acquired and customized 8.3.2 Reactive IS Models 8.3 .3 Environment Model-Based IS 8.3.4 Goal-Based IS 8.3.5 Benefit-Based IS 8.3.6 Learning-Based IS 8.3.6.1 Machine Learning Design 8.3.7 Hybrid IS 8.3. 8 Knowledge-Based (KB) IS 8.3.8.1 Production or Rule-Based KB System 8.3 8.2 Blackboard KB System 8.3.9 SI Models Applied to UbiCom Systems 8.4 Semantic KB IS 8.4.1 Knowledge Representation 8.4 .2 Design issues 8.4.2.1 Open world versus closed world semantics 8.4.2.2 Knowledge life cycle and knowledge management 8.4.2.3 Knowledge creation 8.4 2.4 Knowledge supply and maintenance 8.4.2.5 Design issues for the use of U biCom 8.5 Classical IS logic 8.5.1 Propositional and predicate logic 8.5.2 Argumentation 8.5.3 Design problems 8.6 IS soft computing model 8.6.1 Probabilistic networks 8.6.2 Fuzzy logic 8.7 IS system operations 8.7.1 Search 8.7 .2 Classic (Deterministic) Scheduling 8.7.3 Non-Deterministic Scheduling Exercises References 9 Intelligent System Interaction with Patricia Charlton 9.1 Introduction 9.1.1 Chapter Summary 9.2 Multiplicity of Interactions 9.2.1 P2P Interaction between Multiple Senders and Recipients 9.2.1.1 Unknown Sender and malicious senders 9.2.1.2 Unknown recipients 9.2.1.3 Too many messages 9.2.2 Interaction through intermediaries 9.2.2.1 Access to common communication resources 9.2.2.2 Access to shared computing resources

249 249 250 251 252 252 252 254 255 256 256 257 258 260 260 261 261 263 263 263 265 265 266 266 267 268 268 269 270 271 271 272 272 272 274 275 275 276 276 279 279 279 279 281 281 282 282 283 2833333

content

9.2.2.3 Mediation between applicants and providers Interaction of cooperative participants 9.2.3.1 Coordination 9.2.3.2 Coordination by standards and electronic institutions 9.2.3.3 Hierarchical and role-based organizational interaction 9.2.4 Interaction with interested participants 9.2.4.1 Based on the market interaction and auctions 9.2.4.2 Trading and agreements 9.2.4.3 Consensus-based agreements 9.3 Is interaction design 9.3.1 Making system interaction smarter 9.3.2 Designing interaction between individual intelligent systems 9.3.3 Interaction protocol design 9.3.3.1 Semantics or knowledge-sharing protocols 9.3.3.2 Agent communication languages and language-based protocols 9.3.4 More examples using interaction protocols 9.3.5 Multi-agent systems 9.3.5.1 ACLs and agent platform design 9.3.5.2 Multi-agent system application design 9.4 Some generic intelligent interaction applications 9.4.1 Social media sharing 9.4.2 Referral error and recommendation systems 9.4.2.1 Recommender systems 9.4.2.2 Content-based recommendations 9.4.2.3 Collaborative filtering 9.4.3 Generalized workflow management for humans 9.4.4 Trust management exercises References 9.2.3

10 Autonomous Systems and Artificial Life 10.1 Introduction 10.1.1 Chapter Summary 10.2 Basic Intra-Active Autonomous Systems 10.2.1 Types of Autonomous Systems 10.2.1.1 Intelligent Autonomous Systems 10.2.1.2 Limitation of Autonomous Systems 10.2.2 Auto-* Properties of The Action 10.3 Reflexive and self-aware systems 10.3.1 Self-awareness 10.3.2 Self-describing and self-explanatory systems 10.3.3 Self-modifying systems based on reflective computing 10.4 Self-management and autonomous computing 10.4.1 Autonomous computing design 10.4.2 Autonomous computing applications 10.4.3 Modeling and management of self-star systems 10.5 Complex systems 10.5.1 Self-organization and interaction 10.5.2 Self-creation and self-replication

284 286 287 289 290 291 292 293 295 295 296 297 297 298 300 302 303 304 305 306 307 308 308 308 309 309 309 311 312 317 317 317 318 318 319 319 320 322 322 323 325 326 328 330 331 332 332 335

content

xvi

10.6 Artificial Life 10.6.1 Finite State Automata Models 10.6.2 Evolutionary Computation Exercises References 11 Ubiquitous Communication 11.1 Introduction 11.1.1 Chapter Summary 11.2 Audio Networks 11.2.1 PSTN Voice Networks 11.2.2 Intelligent Networks and Multimedia Subsystems IP 11.2.3 ADLS broadband 11.2.4 Wireless telecommunications networks 11.2.5 Audio transmission networks (radio entertainment) 11.3 Data networks 11.3.1 Network protocol packets 11.3.2 Addressing 11.3.3 Routing and internetworking 11.4 Wireless data networks 11.4.1 Types of wireless networks 11.4.2 WLAN and WiMAX 11.4.3 Bluetooth 11.4.4 ZigBee 11.4.5 Infrared 11.4.6 UWB 11.4.7 Satellite and microwave communications 11.4.8 Roaming between local wireless LANs 11.5 Universal and transparent access to the audio, video and alphanumeric data network 11.5.1 Combined voice and data networks 11.5.2 Combined data and audio-video content distribution networks 11.5.3 On demand 11.6 Pervasive Networks 11.6.1 Wireless Networks 11.6.2 Power Line Communications (PLC) 11.6.3 Personal Area Networks 11.6.4 Body Area Networks 11.6.5 Mobile User Networks 11.6.5.1 Mobile Addresses 11.6 .5.2 Single Path Routing 11.6.5.3 Multipath Routing in Mobile Ad Hoc Networks (MANET) 11.7 Other Aspects of Network Design 11.7.1 Network Access Control 11.7.2 Generalized Versus Localized Access 11.7.3 Access Control Network Access: Firewalls, NAT, and VPN 11.7.4 Group Communications: Multi-Receiver Transmissions 11.7.5 Heterogeneous Interconnection Networks 11.7.6 Global Use: Low-Cost Access Networks for Rural Use 11.7.7 Separation of management and control of use

336 336 337 338 339 343 343 344 344 344 345 346 346 347 347 348 348 349 350 350 352 353 353 354 354 354 355 356 357 358 360 360 360 361 362 362 363 363 364 364 365 365 366 367 368 368 369 369

content

xvii

11.7.8

Service Oriented Networks 11.7.8.1 Service Orientation at the Network Perimeter 11.7.8.2 Content-Based Networks 11.7.8.3 Programmable Networks 11.7.8.4 Overlay Networks 11.7.8.5 Mesh Networks 11.7.8.6 Cooperative Networks

Exercises References 12 Managing Smart Devices 12.1 Introduction 12.1.1 Chapter Summary 12.2 Managing Smart Devices in Virtual Environments 12.2.1 Process and Application Management 12.2.2 Network-Oriented Management 12.2.2.1 FCAPS 12.2.3 Monitoring and Accounting 12.2 .3.1 ICMP 12.2 . 3.2 SNMP 12.2.4 Configuration management 12.2.5 Security management 12.2.5.1 Encryption support for confidentiality, authentication and authorization 12.2.5.2 Protection of the system and its middleware 12.2.5.3 Protection of access devices 12.2.5.4 Protection information 12.2.6 Failure management 12.2 . 7 Performance management 12.2.8 Service-oriented IT management 12.2.8.1 Metrics to assess SOA usage 12.2.8.2 Grid and distributed resource management 12.2.8.3 SLA service management 12.2.8.4 Policy-based service management 12.2.8.5 Generalized workflow Management by Services 12.2.9 Information Management 12.2.9.1 Information Applications 12.2 .9.2 Information rich versus poor and soft versus hard 12.2.9.3 Information explosion management 12.2.9.4 Multimedia content management 12.2. 9.5 Managing Light and Hard Data Using RDBMS 12.2.9.6 Managing Metadata 12.3 Managing Smart Devices in Human-Centric Environments 12.3.1 Richer Management and Softer Data 12.3.2 Service Management Models for Human Users and Environments 12.3.3 Management based on user tasks and activities 12.3.4 Privacy management 12.3.4.1 Biometric user identification 12.3.4.2 Invading technologies n privacy versus privacy-enhancing technologies

370 371 372 372 372 373 375 375 376 379 379 380 380 380 380 382 383 384 384 386 386 388 388 389 391 392 393 394 395 395 396 403 404 404 404 404 404 404 404 404 404 404 404 403 404 404 402 404 402 404 402 404 402 404 404 402 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404 404

content

xviii

12.3.4.3 Entrust regulation of user privacy to service providers 12.3.4.4 Legal approaches to data protection 12.4 Management of smart devices in physical environments 12.4.1 Context-aware 12.4.1.1 Contextual management of physical activities and 12.4.1.2 Context and event management 12.4.2 Micro and nanodevices 12.4.3 Unattended embedded devices Exercises References

411 412 412 412 413 413 415 415 416 416

13 Ubiquitous System: Challenges and Prospects 13.1 Introduction 13.1.1 Chapter Overview 13.2 Challenges Overview 13.2.1 Main Challenges 13.2.2 Multi-tier Support for UbiCom Features 13.2.3 Evolution vs. Revolution 13.2.4 Future Technologies 13.3 Devices 13.3.1 Smaller, more functional smart devices 13.3.2 More seamless assemblies of different devices 13.3.3 Richer system interaction and interoperability 13.3.3.1 Migration of interaction from analog to digital devices 13.3.3.2 Richer digital device interaction 13.4 Intelligent Interaction 13.4.1 Unexpected Connectivity: Accidentally Intelligent Environments 13.4.2 Improvised Service Interoperability 13.5 Interaction of Intelligent Devices in a Physical Environment 13.5.1 Context Awareness: Ill-Defined Contexts vs. a Contextless World 13.5.2 Low Power Consumption energy and sustainable energy consumption 13.5.3 Green UbiCom Devices 13.6 Intelligent Human-Device Interaction 13.6.1 More Diverse Human-Device Interaction 13.6.2 More vs. Less Natural HCI 13.6.3 Analog to Digital and Digital Analogs 13.6.4 Form Follows Function 13.6. 5 Shapes for multifunction devices 13.7 Human intelligence versus artificial intelligence 13.7.1 Posthuman: ICT extends human capabilities beyond the human 13.7.2 Blurred reality and mediated realities 13.8 Social problems: promise versus danger 13.8.1 Increased virtual social interaction versus local social interaction 13.8.2 UbiCom accessible to all 13.8.3 UbiCom affordable to all 13.8.4 Legislation in the digital world and digitization of legislation 13.9 Final remarks Exercises References

421 421 421 422 422 423 424 424 425 425 426 427 427 428 428 428 429 430 430 431 433 436 437 439 439 440 441 441 443 444 444 446 446 447 448 450 451 451 452

Index

455

List of figures

1,1 1,2 1,3 1,4 1,5 1,6 1,7 2,1 2,2 2,3 2,4 2,5 2,6 3,1 3,2 3,3 3, 4 3,5 3,6 3,7 3,8 3,9

Example of a ubiquitous computing application A UbiCom system model Human-ICT Device Interaction ICT Device and Physical World Interaction (CPI) is divided into four interaction subtypes: P2P, P2C, C2P and C2C Three different models of ubiquitous computing: intelligent terminal, intelligent terminal interaction and intelligent infrastructure Some of the main subtypes (triangular relationships) of intelligent devices, environments and interactions and some of their main aggregations (diamond relationships) Alternative views and organizations for the device, environment and interaction entities in the Smart DEI model example for Smart Dust The DataTiles system integrates the advantages of two key interaction paradigms, graphical and physical user interfaces Type of portable computing devices, prototype per man An array of electrodes surgically implanted into Warwick's left arm is monitored and connected to the fibs median nerve nerve strips Audio-Video Cluster distributed in a home local network with a PC as a hub a soft universal local controller for interfacing with washing machine, TV, DVD recorder and radio Different views of ICT system components distributed Abstract view of user access to database and archiving applications The tradeoff between the use of local processing and the required amount of communication depends on the application and how it is designed Different designs for partitioning and distributing information ( I), processing (P) and service access (A) using communication (C) information resources (R ) can be divided within an information system Use of proxies to simplify network access by encoding and decoding transparently the data transmitted on behalf of clients and servers The trade-off of using middleware to reduce the complexity of the ICT system access from to Applications and Types of middleware services to hide Three types of P2P systems, pure, hybrid and partially decentralized The service life cycle: Intelligent services involve a life cycle. Smart Service includes operation and management throughout the entire life cycle

3 10 23 23 26 27 35 47 50 51 52 55 57 78 80 81 82 83 85 86 90 92

3,10 3,11 3,12 3,13 3,14 3,15 3,16 3,17 3,18 4,1 4,2 4,3 4,4 4,5 5,1 5,2 5, 3

5,4 5,5 5,6 5,7 5,8 6,1 6,2 6,3 6,4 6,5 6,6 6,7 6,8 6,9 6,10 7,1 7, 2 7,3 7,4 7,5

List of figures

Service discovery driven by vendors publishing service descriptions Different designs to support distributed interactions asynchronous versus synchronous I/O Two design patterns for intermittent server access, read-ahead and write-behind Shared repositories (left) and event-based interaction (right ) to HLL (High Level Language) is compiled into intermediate (portable) code The main components of an operating system Main functions of the operating system: memory management (MM), process control (PC), inter-process communication (IPC) ) and input/output (IO) control Scheduling multiple tasks that limit the number of available CPUs Example of a thin client-server architecture J2ME uses a virtual machine to support a wide range of devices CPU utilization and power consumption Smart cards and contactless The range of ICT device sizes in use in the 1990s 2000 Using D Rotate, tilt, and stretch gestures to control a screen Human-Virtual Device Interaction, Human-Physical Device Interaction, Physical Human-Human Interaction, which in turn can trigger Human-Virtual Device Interaction Comparison of an approach to conventional functional systems design with a human-device centered design approach. Interactive design requirements consider a broader range of requirements beyond functional and non-functional requirements PVM scenario in Section 1.1.1 Relationship to HCI design heuristics Enabling ubiquitous computing through embedded micro, macro and annotation of physical objects in the world taxonomy for Types and characteristics of tags Application of RFID tags Augmented reality tagging processes A network of sensors used to detect heat gain and report it to a user The main functional characteristics for using a network of sensors Block diagram for a sensor electronic circuit Some examples of MEMS devices, on the order of 10 to 100 microns in size Two simple control systems: a proportional controller (top) and a PID controller (bottom) Using the Lego robot Mindstorm NXt to solve Rubik's Cube Multilevel multidimensional support for a UbiCom property , p. B. Context awareness A conditional scheduling model for context awareness based on pre-planned actions that move the system towards a target context A general architecture for context-aware systems Positioning in mobile networks A composite context (site, person, device and network)-application aware

94 96 98 99 102 107 108 109 110 118 119 121 126 127 139 145

146 148 158 160 165 174 180 182 184 186 188 190 191 195 204 208 218 220 223 228 229

List of figures

7,6 7,7 7,8 8,1 8,2 8,3 8,4 8,5 8,6 8,7 8,8 8,9 8,10 8,11 8,12 8,13 9, 1

9,2 9,3 9,4 9,5 9,6 9,7 9,8 9,9 9,10

10,1 10,2 10,3 10,4 10,5 10,6

Using lateration to determine the position of point O with respect to three reference points A, B, and C Storing and indexing spatial structures in an R-tree to support efficient spatial queries Simple task scheduling for nonpreemptive tasks with execution times, terms, and periods known a priori without resource constraints One-sided active system model (left) vs. two-sided active system and active environment models Intelligent reactive system SI based on environment model according to Russell and Norvig (2003) Two types of SI based goal-based or benefit-based design: basic vs. hybrid, according to Russell and Norvig (2003) Two different learning IS designs Two different designs for a hybrid IS based on horizontal and vertical layers IS design based on a rules-based IS in the know Hybrid IS designs to support UbiCom Two different graphical KRs for device domain One network Bayesian Arb Time modeling of vehicles and passengers arriving and waiting indefinitely at pick-up points Two types of uninformed or brute force search Hierarchical task scheduling and partial task scheduling to view AV content Objective Some examples of intelligent interaction: composition of services, concurrency control for shared resources used, contextual receiver responses, and active mediators acting as filters Some basic examples of multiplicity of interaction Designs for mediators based on who (requester, mediator, or provider) knows what information Multiple SIs configured as one MAS interaction using a set of agent interaction protocols or An agent language The FIPA request interaction protocol Part of the interaction for the plan given in Figure 9.10: Seek help when access to resources fails (left) and delegate the resource access task (right) to a assistant assistant Part of the interaction for th The plan shown in Figure 9.10: seeking advice (left) and negotiating (right) access to resources from multiple resource providers Organizational units (agents) can play multiple roles. Organizational roles constrain the type of interaction A simple planning model for achieving a goal that defines redundant paths through tasks (redundant sequences of tasks) that can be implemented to achieve the goal and that redundant peers can use to implement tasks Architecture of reflective system Three main types of self-internal system* Resource control A high-level schematic architecture for a self-contained computing system that uses managers instead of resources to implement the control loop Control loops to support self-management in various types of systems natural and artificial A finite state machine, represented as a Markov graph for a gate controller Five consecutive generations of Conway's Game of Life show how a sliding pattern in which a shape changes position

xxx

231 235 237 251 253 254 255 257 259 259 260 262 265 271 273 275

280 281 285 299 301 302 303 304 305

306 325 327 328 329 336 337

XIII

List of figures

11.1 Data messages for an application are fragmented into packets D1 to D3 for delivery over different communication networks C1 to C5. 11.2 A typical telecommunications network capable of supporting voice and data over fixed and wireless links. 11.3 A cable video transmission network that also allows the cable provider to act as an ISP 11.4 The difference between mobile and wireless 11.5 An ad hoc network does not have dedicated router nodes 11.6 Data flow, control, and management across the networks Various Layers in a Simplified Network Model 11.7 From Network-Oriented Service Models to Service-Oriented Network Models 11.8 Mesh Networks, Wireless Mesh Networks, and Overlay Networks 12.1 Telecommunications Network Management (TMN) Services and network management (NM) functional areas 12.2 Basic architecture for network management 12.3 V-SAT security model: considerations of security measures that protect system assets against threats 12.4 Some examples of threats using integrated (wireless) networks ) 12.5 Block diagram for a content-based feature detection and identification system 12.6 Classification of us acts ity as a composite context based on a decision tree for individual contexts 13.1 Tiered levels and system support for each of the five main features of the UbiCom system 13.2 The trend towards smaller smart devices with less power and more resources 13.3 The trend 13.4 An Example of Unexpected Connectivity 13.5 The Engineering Process Versus the Reverse Engineering Process 13.6 Human Skill Versus Machine Skill

349 358 359 363 365 370 371 374 382 385 387 390 410 414 423 425

427 429 435 442

table list

1,1 1,2 1,3 1,4 1,5 1,6 1,7 3,1 5,1 5,2 6,1 7,1 7,2 7,3 7,4 8,1 8, 2 8,3 9,1 9,2 9,3 10,1 10,2 11,1 12,1 12,2

Distributed system properties iHCI system properties Context-aware system properties Autonomous system properties Intelligent system properties Comparison of smart devices, smart environments and smart interactions Book chapters and their relationship to the smart DEI model Service access functions used by smart devices UI design heuristics for UbiCom based on high level heuristics proposed by Tidwell (2006) Some examples of lower level HCI design patterns linked to higher level HCI design heuristics based on Tidwell (2005) ) Sensor design and implementation challenges and some corresponding solutions Classification of the main types of context according to the type of environment of the UbiCom system and according to Morse et al. (2000) Different types of context representation according to Strang and Linnhoff-Popien (2004) The main challenges in context modeling Some types of SAS applications with illustrative examples Dimensions into which intelligent systems can be classified Environmental models for UbiCom based systems in the Environment Classification for Intelligent Systems by Russell and Norvig (2003) Intelligent system designs in terms of the types of environments they adapt to Self-star properties for UbiCom systems Greater support for system evolution A comparison of wireless network properties, used for different types of services Intelligent device management requirements FCAPS network management functions

19 19 20 20 20 34 37 76 172 173 190 217 222 225 230 246 248 250 280 287 296 321 323 351 381 383

XXIV

12.3 Relationship between threats, assets and protection measures from the perspective of a mobile smart device user 12.4 Seven different models for a user-centric service management 12.5 Different types of biometric identification 13.1 Challenges in the design of the support of the properties of the UbiCom system 13.2 Past, present and future UbiCom interaction 13.3 Comparison of specific behaviors of humans and intelligent systems

table list

388 405 409 422 438 442

Foreword Ubiquitous computing, often referred to as ubiquitous computing, is a vision for computing systems to permeate the physical world and human and social environments. It is concerned with making computers more physical, in the sense of developing a broader variety of computing devices that can be usefully implemented in more physical environments. It is concerned with the development of situated and ubiquitous technology that is easily accessible and usable by humans and that can be designed to work harmoniously in human and social settings.

Target Audience This book is primarily intended for computer scientists and technologists in education and industry, so that they can keep up with the latest developments in a diverse field of computer science in one text. The aim is to promote a much more interdisciplinary exchange of ideas within the sub-areas of computing and between computing and other related areas. In essence, it links various subfields of computing, distributed computing, communications networks, artificial intelligence, and human-computer interaction, explaining and extending designs involving mobile services, service-oriented computing, sensor networks, microelectromechanical systems, context-aware computing. . , embedded systems and robotics, as well as new developments on the Internet and the web. This is a good text for modeling in these areas. The most important prerequisite for understanding this book is a basic understanding of computing and technology. Parts of the book should be easily understood by middle and final grade computer science students, although parts can also be used as an introductory textbook to highlight some of the amazing things that are happening in the world of systems. of ICT. It is also suitable for master's level students and for interdisciplinary use in courses that only include computer science as part of the course. The author hopes that this text will help stimulate the interest of a broader audience in some of the advanced ideas in computing and lead to the development of new courses in advanced computing. An overview of the book can be found at the end of the first chapter.

Teaching with this book The website of the author of the book is available at http://www.elec.qmul.ac.uk/people/stefan/ubicom. The website contains PowerPoint slides for the book, additional exercises and selected solutions to the exercises, an online bibliography for the book, etc. The book's website also provides tips on using this book in various types of educational courses and training programs.

Gracias

Patricia Charlton, Michael Berger, and Robert Patton were involved in this book project from the beginning. Patricia Charlton in particular contributed many good ideas, especially in the AI chapters, two of which she co-authored. Several international colleagues provided comments on specific sections of the book: Barbara Schmidt-Belz, Heimo Laamanen, Jigna Chandaria, and Steve Mann; as well as several QMUL colleagues: Athens Ma, Chris Phillips, Karen Shoop and Rob Donnan. The content of this book arose in part from teaching various undergraduate and master's courses in distributed computing, AI, HCI, and other applied computer science at various universities, but particularly from teaching ELEM038, Mobile Services, to students at Queen Mary, University of London. Second, this book grew out of research in the following projects: AgentCities, CASBAH, Context-aware Worker (an EPSERC Industrial Case Award project with BT, John Shepherdson), CRUMPET, EDEN-IW, iTrust, My e- Director 2012 and working with my research assistants: Uko Asangansi, Ioannis Barakos, Thierry Delaitre, Xuan (Janny) Huang, Kraisak Kesorn, Zekeng Liang, Dejian Meng, Jim Juan Tan, Leonid Titkov, Zhenchen Wang and Landong Zuo. Some of them helped verify parts of this text. Birgit Gruber started this book project at Wiley. Sarah Hinton and Anna Smart guided the book through the various stages of design to the finished product, while Susan Dunsmore and Sunita Jayachandran helped put the finishing touches. Ultimately, my family was very supportive and encouraged me to finish it.

1 Ubiquitous Computing: Fundamentals and Vision 1.1 Living in a Digital World We live in an increasingly digital world populated by a plethora of digital devices designed to support and automate more human tasks and activities, human social interaction enriches and enhances physical interaction in the world1. The physical environment is increasingly digitally instrumented and littered with embedded sensor-based and control devices. These can recognize our location and automatically adapt to it, which facilitates access to localized services, e.g. B. Open doors and turn on lights when we approach them. Positioning systems can determine our current location as we move. They may be linked to other information services, e.g. B. suggest a map with a route to our destination. Devices such as contactless cards and keys can be used to gain access to protected services in the area. Epaper2 and e-books allow us to wirelessly download up-to-date information on flexible digital paper without having to go to a physical bookstore. Even electronic circuitry can be distributed wirelessly to dedicated printers, allowing electronic circuitry to be printed on a paper-like substrate. Wired and wireless networks at speeds of megabits per second exist in many parts of the world for the transmission of multimedia content (alphanumeric text, audio and video) in the workplace and home and for use by mobile users and fixed locations. The increasing use of wireless networks allows more devices and infrastructure to be inserted into the physical environment in a piecemeal and less disruptive manner. Electronic circuits and devices can be made smaller and cheaper, and operate more reliably and with less power. There are a plethora of smart multipurpose mobile devices out there.

The physical world is often referred to as the real world or environment to distinguish it from both perceived human views of the world (imaginary worlds) unrelated to reality and basic facts, and from computer-generated world views (virtual worlds). . 2 A distinction should be made between digital hardware versions of analogue objects, e.g. ePaper and digital or electronic copies of information contained in analogous objects, e.g. E-tickets, for airlines. The latter type is known as vTickets, short for virtual tickets.

Stefan Poslad

Ubiquitous Computing: Fundamentals and Vision

Access to local and remote services. Mobile phones can act as multiple cameras and audio and video players, as information devices and game consoles.3 Interaction can be personalized and dependent on user context by sharing personalization models on our mobile devices with other services when we interact with them, p. , audio and video devices can be pre-programmed to display only a person's preferred content options. Many types of services that support everyday human activities related to food, energy, water, distribution and transportation, and health are highly dependent on computers. Traditionally, service access devices have been designed and targeted at human users performing activities that access isolated individual services, e.g. B. we access information or watch videos or talk on the phone. In the past, if we wanted to access multiple services and combine them to support multiple activities, we had to use separate access devices. In contrast, current service offerings can provide more integrated, interoperable and ubiquitous service delivery, e.g. B. the use of data networks to also offer video transmissions and voice services, which is called provision of triple play services. There is plenty of scope to develop them further (Chapter 2). The term "ubiquitous", meaning that it occurs or is present everywhere, combined with computing in the term ubiquitous computing (UbiCom) is used to describe the ICT (information and communication technology) systems that make possible make information and tasks available everywhere and to support intuition. human use that appears invisible to the user.

1.1.1 Chapter Summary To support the understanding of ubiquitous computing, this introductory chapter describes some illustrative applications of ubiquitous computing. Next, the proposed holistic framework at the heart of UbiCom called the Smart DEI (pronounced smart 'day') UbiCom Framework will be introduced. It will first be looked at from the perspective of UbiCom's core internal properties (Section 1.2). Next, UbiCom is considered from the external interaction of the system through the core system environments (virtual, physical and human) (Section 1.3). Third, UbiCom is considered in terms of three basic architectural blueprints or "design patterns": smart devices, smart environments, and smart interaction (Section 1.4). The name of the DEI framework is derived from the first letters of the terms Devices, Environments, and Interaction. The last major section (Section 1.5) of the chapter describes how the entire book is organized. Each chapter concludes with exercises and references.

1.1.2 Illustrative Applications of Ubiquitous Computing The following applications in the human and physical environment illustrate the range of benefits and challenges of ubiquitous computing. A personal reminders scenario focuses on users recording audio and video content, automatically recognizing the user's context and annotating the recordings. A 21st Century Planned Traffic Scenario focuses on schedules, adapting your preset plans to the actual state of the environment and further disseminating this information. A food management scenario focuses on how non-electronic analog objects such as food can be digitally connected to a computer system to control their use by humans. A fully automated food management system could include robots that can move physical objects and quantify the level of a series of analog objects. A utility management scenario

And of course, there's nothing to stop this from happening the other way around: game consoles can act as phones, audio and video players and recorders, etc., and cameras can act as phones, etc.

Living in a digital world

focuses on how to connect electronic analogue devices to a UbiCom system and manage their usage in a user-centric way, allowing them to work together to achieve common goals.

1.1.2.1 Personal memories As a first motivating example, consider recording a personal memory of the physical world (see Figure 1.1). Until about the 1980s, before the advent of the digital camera, photography consisted of manually shooting a light meter and then manually adjusting the camera's aperture and shutter speed relative to the light meter, so that the exposure on light-sensitive chemical film was correct.4 This involved manually focusing the camera's lens system. The camera film behaved like a sequential recording medium: a new recording requires rolling the film to the next blank section. It had to wait until the entire film of a series of frames, usually 12 to 36, had finished before sending the recorded film to a specialized film processing company with special equipment to convert the film into a special format that could be seen. Making additional prints would also require the services of a specialized film processing company. A digital camera automatically captures an image of a part of the physical world scene on a built-in display. When using digital cameras, taking pictures is much less intrusive to the subject than using film cameras.5 The camera can automatically focus and expose recorded images and

The GPS transmitter is part of the UbiComp system

positioning

image processing clock

announcement

AV Capture AV Player

AV-Datenbank

Projector Communication Removable Storage Printer

Automatic face detection, recognition, etc.

DD/MMAA x,y

SAY

Figure 1.1 Example of a ubiquitous computing application. AV recording is personal, location based (via GPS), time based and networked to interact with other ICT devices such as printers and a database of family and friends.

4 There was an easier to use camera called a point and shoot that used fixed focus (set to infinity) and exposure for film. 5 For example, digital cameras are less intrusive: they reduce the need for a photographer to ask the subject to say "cheese" to focus the subject in the camera, since it is easy and inexpensive to capture an entire series of photos in one collects and quickly removes those that are not considered aesthetic.

Ubiquitous Computing: Fundamentals and Vision

Video, so that recordings are automatically focused and selected parts of the scene are optimally lit. The context of the recording, such as B. Location and date/time are also automatically recorded by the built-in location and clock systems. The camera is aware that the person taking a picture may be interested in bringing people into focus in a scene, even if they are not in the center. It does this through an enhanced user interface, where markings on facial parts, such as eyes and mouth, are automatically superimposed on the view of the physical world, either on an integrated display or through a lens or viewfinder. The lens is then automatically focused so that the faces in the visual recording are sharp. Recorded content can be instantly viewed, printed and shared with friends and family using removable storage or exchanged over a communications network. It can be archived in an external audiovisual (AV) content database. When AV content is saved, it is tagged with time and place (the GIS database is used to convert position into a place context). Image processing can be used to perform facial recognition to automatically tag anyone who can be recognized in the database of friends and family. By using microelectromechanical systems (MEMS (Section 6.4)), a separate device the size of a decimeter, e.g. B. a projector can be installed. The camera is networked and has the ability to discover other specific types of ICT B. printers to enable printing from the camera. Network access, music and video player and video camera functions can also be combined in this single device. Ubiquitous Computing (UbiCom) encompasses a wide range of computers, not just devices that are general purpose computers,6 multifunctional ICT devices such as telephones, cameras and game consoles, ATMs (cash dispensers), vehicle control systems, mobile phones, electronic devices calculators, household appliances and computer peripherals such as routers and printers Embedded systems (computers) are characterized by the fact that they are autonomous and perform certain predefined tasks, which allows designers to optimize them as follows There are fewer need for full OS functionality, eg B. Multi-process scheduling and memory management, and less need for full CPU, eg. B. the simple 4-bit microcontrollers used to play a melody on a greeting card or children's toy. This reduces the size and cost of the product, allowing it to be mass-produced more cheaply and benefit from economies of scale. Many objects could be designed to be an AV capture compatible multifunction device, AV player, communicator, etc. Embedded computer systems may be subject to a real-time limitation, embedded real-time systems, e.g. B. A vehicle's anti-lock brakes may have a real-time restriction that the brakes must be released in a short period of time to prevent the wheels from locking up. ICT systems are becoming increasingly complex as we dynamically connect a greater variety and number of individual systems in multiple ways. In some cases, for ICT systems to become more useful, they need to be more closely linked to their global physical environment; h should be contextually aware of their local physical environment. In order to make ICT systems more usable by humans, ICT systems must find the right balance between autonomous action and human action. Currently, it is not possible to completely eliminate humans from the cycle in the development and maintenance of the operation of significantly complex systems. ICT systems should be designed in such a way that the responsibilities of automated ICT systems are unclear and the responsibilities of human designers, operators and maintainers are clear, and in such a way that human cognition and behavior are not overloaded.

Of the billions of microprocessors manufactured each year, less than 5% find their way into multi-application programmable computers, and the other 95% or so are used in a variety of applications and embedded systems.

Living in a digital world

1.1.2.2 Adaptive regular service In a 20th century regular service, the schedules of a regular service, e.g. B. Taxi, bus, train, plane, etc. to pick up passengers or merchandise at fixed or scheduled points, only accessible in terminals and special places. Passengers and controllers have a limited view of the real time when vehicles arrive at specific points on the route. Passengers or goods may arrive at designated collection points and wait for a long time. A manual system allows drivers to radio their current position to traffic controllers in the event of time deviations. Air traffic controllers can often only manually inform passengers of delays at designated pickup points. In contrast, in a 21st century scheduled transportation service, the position of the transportation vehicles is determined using automated positioning technology, e.g. B. GPS determined. For each vehicle, the time it takes to reach specific collection points, e.g. B. next stop, last stop, estimated in part based on the vehicle's current position, progress, and historical data from route users. The current arrival times of the vehicles can be consulted ubiquitously via mobile phones, which allows JIT (just in time) arrival at the collection points of people and goods. Vehicles on the route can mark locations that they expect to change the time of other vehicles in that area. The probable places of time change can be verified by all the following vehicles. Vehicles can then be dynamically rerouted and rescheduled based on "time change" locations, current positions, and service demand. If the capacity of the transport vehicles were expandable, the volume of passengers on the route could determine the capacity of the transport service to meet the demand. The transportation system may have to deal with conflicting objectives, such as B. accommodating more passengers and freight to generate more revenue for the services provided, rather than minimizing vehicle delay at pre-arranged points along its route.

1.1.2.3 Food management A ubiquitous home environment is designed to support healthy eating and weight management for food consumers. A conventional system does this manually. A next-generation system (semi-)automates this task with connected physical devices, such as refrigerators and other food and beverage storage areas, that can control the entry and exit of food. Sensors are integrated into the system, e.g. B. to determine the weight of food and people. The scanners can scan food and drink packaging for barcodes, text tables, expiration dates, and food ingredients and weight percentages. Integrated handheld scanners can also select foods to buy at grocery stores, such as supermarkets, that should be avoided for personal or health reasons. The system recognizes who buys what groceries in the supermarket. The system allows automatic configuration of food recipes to match the available ingredients. Grocery stocks can be monitored periodically to alert users when groceries are expiring and essential grocery supplies are running low. The amount of food consumed per unit of time and per person can be monitored at different levels of granularity in terms of total amount of food and in terms of weight in grams of fat, salt and sugar, etc. The system may include guidelines for following a balanced diet, e.g. B. Consume five pieces of fruit or vegetables a day. The system design includes the following components. Scanners are used to identify the type and amount of ingredients in the package. This can include a barcode, but not all foods can be barcoded and identified in this way. Home grocery store can be designed to check when (selected) groceries are running low. You can define run out of food when the amount of an item remains, but items can be large and partially full. The amount of food remaining must be measured with a weight converter, but the weight of the container is needed to calculate the weight of the food. The local grocery store might

Ubiquitous Computing: Fundamentals and Vision

programmed to detect when food has expired by reading the expiration date and indicating that the food is inedible. Many exceptions or conditions may need to be specified for the system to handle food storage. For example, food may still be edible after its expiration date has passed. Foods that are frozen and then thawed in the refrigerator may be out of date but are still edible. Selected system events could trigger actions automatically, e.g. B. Small amounts of groceries may trigger actions to automatically buy groceries and receive them. Operating policies should be linked to the context or situation and the authority to act on behalf of the owner, e.g. B. Groceries are not ordered when consumers are away, or consumers indicate they do not want infinite orders of expired or near-empty groceries. There may be limitations with full system automation. Unless the system can act on behalf of the human owner to receive the delivery, allow physical access to the grocery store at home and the building where consumers live, and have robots to move physical objects and the opening and closing of the home grocery store to service you. in temperature controlled environments there, these scenarios require some human intervention. An important issue in this scenario is the trade-off between what the system can and should do and what humans can and should do.

1.1.2.4 Consumption regulation A ubiquitous home environment is designed to regulate the consumption of a consumable (such as water, energy or heating) and improve the efficiency of use. For example, supply management products, e.g. B. Manually configurable by human users, power management products use independent devices and are designed to recognize changes in the local user context. User context aware power devices can be designed to turn on in a specific way, e.g. B. a light turns on, a heater turns on when it detects the presence of a user, otherwise it turns off. These devices must also take environmental conditions into account so that artificial lighting and heating are not turned on when natural lighting and heating levels are determined to be sufficient. The system design includes the following components and usage patterns. Manually configured devices can waste power because users forget to turn them off. Devices that are set as active according to preset user policies, e.g. B. Controlling a timer can waste energy because users cannot always schedule their activities to meet the static schedule of the timer. Individually context sensitive devices such as lamps can waste energy as multiple overlapping devices can be activated and turned on, e.g. B. when the presence of a user is detected. A ubiquitous system can be designed using multi-agent systems and autonomous system models to function as an intelligent network. Multiple devices can self-manage and work together to comply with user policies, e.g. B. Minimization of energy consumption. If, for example, several overlapping devices are classified as redundant, the system decides which one should be powered on. The cost of power consumption depends on several factors, not only on the time that a device is on, but also on the energy rating, which varies depending on the device and the tariff, ie. h the cost of energy consumption varies depending on the time of day. Advanced usage meters can be used to show usage per unit of time and per device and can allow customers to see how they are using energy and manage their usage more efficiently. Demand response designs can adjust power consumption in response to dynamic pricing signals and policies. For example, during peak periods when prices are higher, power-hungry devices might run more sparingly to save money. A direct load response system, a type of demand response system, can also be used in which the electric provider or a third party remotely controls certain devices that use customer power during peak load periods. More examples of applications for ubiquitous computing are discussed in Chapter 2.

Living in a digital world

1.1.3 Holistic Framework for UbiCom: Smart DEI Three approaches to analyzing and designing UbiCom systems are proposed to form a holistic framework for Ubiquitous Computing, called the Smart DEI7 Framework, based on:

• Design architectures for applying UbiCom systems: three main types of design for UbiCom systems •

•

suggested: Smart Device, Smart Environment and Smart Interaction. These statements are described in more detail in section 1.4. An internal model of the UbiCom system properties based on five fundamental properties: distributed, iHCI, context-aware, autonomous, and artificial intelligence. Depending on the degree to which these five properties are supported and linked to one another, there are many possible subspecies of ubiquitous system design. This model and its properties are described in Section 1.2. A model of the interaction of the UbiCom system with its external environments. In addition to a traditional distributed ICT system-device interaction within a virtual environment8 (C2C), two other types of interaction stand out: (a) between computer systems and people as systems (HCI); (b) between computers and the physical world (CPI). Environmental interaction models are described in Section 1.3.

Smart devices, e.g. Mobile smart devices, smart cards, etc. (Chapter 4) are more focused on interaction within a virtual (computer) world and are less aware of the context of the physical world compared to ambient smart devices. Smart devices tend to be less autonomous, as they often need to directly access external services and act as personal devices that are manually activated by their owner. There is more emphasis on the design of these devices to take into account the human context of use. They may contain certain types of artificial intelligence, e.g. B. Machine vision allows cameras to recognize human face elements in an image, e.g. B. based on eye and mouth recognition. Intelligent environments consist of devices such as sensors, controllers, and computers that are embedded in or operate in the physical environment, e.g. B. Robots (Section 6.7). These devices are highly aware of the context of their physical environment in relation to their tasks, e.g. B. A robot must grasp and model the physical world to avoid obstacles. Smart environment devices can be aware of certain user activities, e.g. B. Doors that open when people walk towards them. They often act autonomously without manual guidance from users. These include certain types of intelligence, e.g. B. Robots can create complex models of physical behavior and learn to adjust their movement based on experience. Intelligent interaction focuses on more complex models of interaction of distributed software services and hardware resources, dynamic collaboration, and completion among multiple entities on multiple devices to achieve the goals of individual entities or achieve a collective goal. For example, a smart camera could work with smart lighting in a building to optimize the lighting to capture an image. Multiple lighting fixtures in a physical space can work together to optimize lighting and still minimize overall energy consumption. Intelligent interaction focuses less on awareness of the physical context and more on user contexts, e.g. B. User goals, such as the need to reduce overall value

Smart DEI stands for the Smart Devices, Environments and Interactions model. Pronounced "Smart Day" to indicate that the model is focused on using system support for daily activities. 8 A virtual (computing) environment comprises the common distributed ICT infrastructure in which the individual applications of the UbiCom system are operated. Also note that there are other subtypes of virtual environments such as B. Virtual reality environments in which human users can interact with computer simulations of slices of imaginary worlds using multi-modal sensory interfaces.

Ubiquitous Computing: Fundamentals and Vision

Power consumption of all devices. Intelligent interaction often uses distributed artificial intelligence and multi-agent system behaviors, e.g. B. Hire web interaction to propose tasks. The Smart DEI model represents a holistic framework for building different UbiCom systems based on smart devices, smart environments and smart interaction. These three layout types can also be combined to support different types of smart spaces, e.g. B. Smart mobile devices can combine location awareness of their changing physical environment to optimize routing of physical assets or computing environment from another location. All smart devices are networked and, as a main feature, they can exchange data and access information services. A comparison of a type of Smart Device, Smart Environment and Smart Interaction (see Table 1.6) is also made later in relation to its main properties of the UbiCom system of Distributed, Context-Aware, Intrusive HCI, Autonomy and Intelligence and in relation to the types of the physical world, human interactions and the ICTs that support them.

1.2 Modeling the key properties of ubiquitous computing A world in which computers disappear into the background of an environment made up of intelligent rooms and buildings was first articulated more than fifteen years ago in a vision called ubiquitous computing by Mark Weiser ( 1991). Ubiquitous computing represents a major shift in computing, where people live, work, and play in an integrated computing environment that is nested in the world. Ubiquitous computing posits a world where people are surrounded by computing devices and a computing infrastructure that supports us in everything we do. Traditional networked computer systems10 or information and communication technology (ICT) systems see themselves as set in a virtual world or environment of other ICT systems and form a system of ICT systems. Computer systems behave like distributed computer systems that are linked through a communications network. In conventional ICT systems, the role of the physical environment is limited, for example, the physical environment acts as a channel for electronic communication and power and provides the physical resources to store data and execute electronic instructions, thus supporting an ICT environment. virtual. Due to the complexity of distributed computing, systems often project varying degrees of transparency to their users and providers to hide from users the complexities of the distributed computing model, such as who to send, what to send, and not where. Traditionally, human-computer interaction (HCI) with ICT systems has been structured with a few relatively expensive access points. This mostly uses keyboard and pointing device input, which is quite intrusive to interact with. Weiser's vision centers on digital technology that is interactive but more discreet and ubiquitous. His main concern was that the computer interfaces would get too much human attention. Unlike good tools that become an extension of ourselves, computers often don't allow us to focus on the task at hand, instead distracting us from figuring out how to make the tool work properly. Weiser used the analogy of writing to explain part of his vision of ubiquitous computing. Initially, writing required experts, such as scribes, to produce the ink and paper used to present the information. Only additional experts, such as scientists, could understand and interpret the information. Today printed text (created and formatted with computers) on paper and electronic copy

Here, a computer is considered to be any device, simple or complex, small or large, that is programmable and has memory to store data and/or code. 10 A system is defined at this stage as a set of interconnected components of interest. Systems are often a system of systems. Anything outside of system boundaries is the system's environment.

Modeling of the most important ubiquitous computing properties

The text displayed on computer-based devices is very ubiquitous. Of both, printed text is even more prevalent than computer text.11 In many parts of the world, most people can access and create information without conscious thought about the processes involved.12 More visions of ubiquitous computing are discussed in Chapters 2 and 3. discussed in the last chapter (Chapter 13).

1.2.1 Main characteristics of UbiCom systems The characteristics that distinguish UbiCom systems from distributed ICT systems are as follows. First, they reside in custom human-centric environments and interact in a less intrusive way with humans. Second, UbiCom systems are part of and used in physical environments because they capture more of the physical environment. Because they are more aware of it, they can adapt to it and are able to react and control it. Thus, Weiser's vision for ubiquitous computing can be summarized in three basic requirements: 1. Computers must be networked, distributed, and transparently accessible. 2. Human-computer interaction needs to be hidden more. 3. Computers must take context into account to optimize their performance in their environment. It is proposed that there are two additional basic types of requirements for UbiCom systems: 4. Computers can operate autonomously, without human intervention, self-managed, as opposed to pure human-computer interaction (point 2). 5. Computers can handle a variety of dynamic actions and interactions, driven by intelligent decision making and intelligent organizational interaction. This may involve some form of artificial intelligence to handle: (a) incomplete and non-deterministic interactions; (b) cooperation and competition between members of organizations; (c) richer interaction by sharing context, semantics and objectives. Therefore, an extended model of a ubiquitous system is proposed. These two additional behaviors allow ubiquitous systems to operate in additional environments. These environments fall into two groups: (a) human-centered, personal, social, and economic environments; and (b) physical environments of living things (ecologies) and inanimate physical phenomena. These five UbiCom requirements and the three types of environments (ICT, physical and human) are not mutually exclusive, they overlap and need to be combined.

1.2.2 Distributed ICT systems ICT systems are by nature distributed and networked. Multiple systems often behave like a single system and appear to the user as a single system, i. h various systems are transparent or hidden from the user. Individual systems can be heterogeneous and can be connected and disconnected from the ICT system infrastructure at any time: openness.

11 Many people thought that the rise of computers would lead to a paperless world, but that hasn't happened yet, see exercises. 12 Keep in mind that the ability to read and write texts and understand an average vocabulary currently requires several years of training. Handwriting inherently depends on natural language interfaces, which have fundamental limitations.

Ubiquitous Computing: Fundamentals and Vision

Physical Ecological Environments (Living) CPI

human environments

physical phenomenon

personal

CPI (Perceive, Adjust)

(Survival, Adaptation)

HCl

Advertising

social

HCI (complete)

HCI (cooperate)

implicitHCI Autonomous Distributed

contextual awareness

Intelligent

System UbiComp

ICT CCI Virtual ICT Environments

Figure 1.2 A UbiCom system model. The dotted line shows the limit of the UbiCom system

1.2.2.1 Networked ICT Devices Ubiquitous computers are networked computers. They offer services that can be accessed locally and remotely. In 1991, Weiser considered that ubiquitous access through "transparent links of wired and wireless networks was an unsolved problem". Since then, however, both the Internet and cellular wireless networks have evolved to provide seemingly ubiquitous network access. Various communication networks exist in terms of reach, performance, content, topology, and design (Chapter 11) to support the UbiCom interaction.

1.2.2.2 Transparency and openness Buxton (1995) saw the core of Weiser's vision of ubiquitous computing as ubiquity (access is everywhere through different devices) and transparency (access is hidden, embedded in the environments), but these seem to represent something obvious, paradoxically, how can something be everywhere and yet be invisible? The point here is not that technology cannot be seen (heard or touched), but that its presence does not invade the work environment, either in terms of the physical space or the activities that are carried out. This description of transparency is strongly linked to the idea that devices and functions are embedded and hidden within larger interactive systems. Also note that the vision seems to imply a binary classification of the transparency of the system, moving from no transparency to full transparency. In practice, the transparency of the system is often less clear. Systems may have partial connectivity and limited ability to interoperate with their environment, making it difficult to support transparency. The properties of ubiquity and transparency are central characteristics of distributed system types. One last key property of distributed systems is openness: open distributed systems. Openness allows systems to avoid having to support all of their features at design time, thus avoiding a closed implementation. Distributed systems can be designed to support different degrees of openness

Modeling of the most important ubiquitous computing properties

Dynamically discover and access new external services. For example, a UbiCom camera can be configured to detect printing services and notify users that they are available. The camera can then transmit its data to the printer for printing. Openness often leads to complexity and reduces availability. When one feature is active, others may need to be disabled, e.g. B. Some devices cannot register one input while showing another output. Openness can introduce heterogeneous functions into a system that are incompatible and make the entire system unavailable. Openness can reduce availability because operations can be disrupted as new services and features are rolled out. Note that many systems are still designed to limit openness and interoperability, even when there appears to be great benefit to not doing so. For example, messages stored on most home answering machines cannot be easily exported for review or as part of a conversation with others. It would be very easy to design phones to share their content via plug-and-play removable media and a wireless network, and to make them more configurable so that users can customize the amount of message storage they need. Providers can intentionally and selectively reduce the opening, e.g. B. transparently ignore the presence of another competitor's services in order to maintain its market share. Distributed ICT systems are often designed in the form of a layered model, comprising: (1) a layer of hardware resources at the bottom, e.g. B. Data source, storage and communication; (2) middleware and operating system services in between, e.g. B. to support data processing and data manipulation; and (3) a human-computer interaction layer on top. This layered ICT model oversimplifies the UbiCom system model, as it does not model heterogeneous patterns of system interaction. This ICT model generally involves only a simple explicit human interaction and a simple physical world interaction model. Distributed computing systems are covered in most chapters, but especially in Chapters 3, 4, and 12. Chapter 11 covers your communications infrastructure.

1.2.3 Implicit Human-Computer Interaction (iHCI) Many human-device interactions are designed to support explicit human-computer interactions expressed at a syntactically low level, eg. B. to activate specific controls in that specific order. Also, as more tasks are automated, device diversity increases and more devices must work together to accomplish tasks. The high volume of explicit interaction can easily annoy, distract, and overwhelm users. Interactive systems should be designed to support higher levels of implicit human-computer interaction, or iHCI (Chapter 5).

1.2.3.1 The silent computer The concept of the silent or vanishing computer model has several dimensions. It may mean that programmable computers as we know them today will be replaced by something else, e.g. B. human brain implants that are no longer physically visible. It can mean that the computers are there but hidden, e.g. B. when it comes to implants or miniature systems. Alternatively, the disappearance of the computer focus may mean that computers are not really hidden; They are visible but imperceptible since they are part of the peripheral senses. They go unnoticed due to the effective use of implicit human-computer interaction. The forms and modes of interaction that allow computers to disappear will depend in part on the target audience, as social and cultural boundaries may have different profiling attributes in terms of technology drivers. For some groups of people, ubiquitous computing is already here. Some people consider apps and technologies, such as mobile phones, email, and chat messaging systems, a necessity to function on a daily basis.

Ubiquitous Computing: Fundamentals and Vision

Some researchers see the promise of ubiquitous computing as a technology that resolves itself into behavior and invisibly permeates the natural world as unattainable. B. Rogers (2006). Several reasons are given to support this view. The widespread use of quiet computing drives people away from being proactive: systems are proactive rather than people. Quiet computing is a computationally intractable problem when used generally and ubiquitously. Because technology is inherently artificial, it separates the artificial from the natural. What is considered natural is subjective and cultural and to some extent technological. This blurs the distinction between the means to directly reshape nature at the molecular level and the means to influence nature at the macro level, e.g. B. Pollution and global warming (Chapter 13). The intrusiveness of technology depends in part on the user's familiarity and experience with it. Alan Kay13 is credited with saying: "Technology is anything that is invented after you are born." Everyone sees technology as something invented before they were born. When Calm Computing is used in deterministic environments, in environments with constrained applications, in a narrower sense, and supported at multiple levels depending on the needs of the application, it becomes second nature14; then, Calm Computing models can be successful.

1.2.3.2 Implicit vs. Explicit Human-Computer Interaction UbiCom's original vision focused on making computing and accessing digital information more seamless and less intrusive. Achieving this in part requires that systems not require users to explicitly specify every detail of an interaction to complete a task. For example, the initial use of many electronic devices requires users to explicitly configure some proprietary controls of a timer interface. It should be implied that if devices use absolute times to schedule actions, then when the device is first used, the time should be set. This type of implicit interaction with the computer is called implicit human-computer interaction (iHCI). Schmidt (2000) defines iHCI as “an action performed by the user that is not primarily intended to interact with a computerized system, but is understood as input by that system”. To reduce the level of explicit interaction with computers, several factors must be carefully considered. It requires users to be comfortable relinquishing increasing control to automated systems that continue to invade their lives, perhaps without the user realizing it. It requires systems to be able to reliably and accurately recognize the user and the context of use and adjust their operation accordingly.

1.2.3.3 Embodied Reality versus Virtual, Augmented and Mediated Reality Reality refers to the state of actual existence of things in the physical world. This means that things exist in time and space as experienced through a conscious sense of human presence and are located and embodied in the physical world. Technology can alter the human perception of reality in a variety of ways, including virtual reality, augmented reality, mediated reality, hyperreal reality, and telepresence (Section 5.4.4). Virtual Reality (VR) immerses people in a continuous, unembedded, computer-generated world. Virtual reality is often generated by a single system where time and space coincide and exist as one.

Working at the Xerox Corporation's Palo Alto Research Center (PARC) in the 1970s, Kay was one of the key researchers who developed the first prototypes of network workstations later marketed by Apple, i. h the Apple Macintosh. 14 Second nature is an acquired behavior that has been practiced for so long that it seems innate.

Modeling of the most important ubiquitous computing properties

Separate reality from the physical world. Augmented Reality (AR) is characterized by immersion in a physical environment, where physical objects can be linked to a virtual environment. AR can enhance physical reality by adding virtual views to it, for example using various techniques such as transparent screens and homographic views. Augmented Reality can be viewed from both the HCI perspective (Section 5.3.3) and from the physical world interaction perspective (Section 6.2). While in augmented reality computer information is added to enhance real world experiences, in the more general nature of the mediated reality environment,15 reality can be reduced or modified as desired. An example of altering reality rather than augmenting it is instead of wearing glasses to correct personal vision deficiencies, using them to mask far-field vision to focus on near-field tasks. Weiser made a comparison between VR and UbiCom, calling UbiCom the opposite of VR. Unlike virtual reality, ubiquitous computing brings the use of computers into the physical world with humans. In fact, the contrasting notion of ubiquitous and invisible computing versus virtual reality is so strong that Weiser coined the term "embedded virtuality." He used this term to refer to the process of "taking computers out of their electronic shell." In this text, the term "device" is used to focus on the concept of embedded virtuality rather than the broader notion of a virtual service. Various devices can also form systems of devices and systems of systems. In highly open virtual systems, data and processes can exist anywhere and be accessible from anywhere, leading to loss of (access) control. The potential for data breaches is increasing. In physical and virtual embodied systems, such effects are reduced by the implicit limitations of the embodiment. Incarnated virtuality has multiple connotations. For computers to be used most effectively in the physical world, they can no longer remain embedded in limited electronic forms like the personal computer, but must exist in a broader range of forms that must be more ubiquitous, flexible, and stationary. Hence Weiser's emphasis on explicitly mapping a broader spectrum of everyday computing devices in the form of tabs, pads, and boards (Section 1.4.1.1). Distributed computing works through its increasing ability to work together seamlessly to create a virtual computer out of a group of individual computers; it hides the detailed interaction with the individual computers, and it hides the realization within individual forms that form a virtual realization for computing. The use of many different physical mechanisms (including chemical and biological) and the virtual assembly and reassembly of nature at different levels can also change the essence of what human nature and naturalness are (Sections 5.4, 13.7). With an increasing reliance on embedded virtual computers, UbiCom, humans are also at risk of extinction from the incarnation (Hayles, 1999).

1.2.4 Context awareness The goal of UbiCom systems is not to support global ubiquity, the networking of all systems in a ubiquitous service domain, but to support context-based ubiquity, eg. B. situated access versus mass access. The benefits of context-based ubiquity include: (1) limiting the resources needed to provide ubiquitous services, as providing ubiquitous services would be prohibitively expensive; (2) restrict the choice of access from all possible services to only useful services; (3) avoid overloading the user with too much information and decision making; and (4) support a natural focus of attention and calm decision-making by users.

quince

Reality can also be modified by many other mechanisms, not just virtual computing mechanisms, e.g. B. chemical, biological, psychological, etc.

Ubiquitous Computing: Fundamentals and Vision

1.2.4.1 Three main types of environmental contexts: physical, user, virtual There are three main types of context awareness16 for external environments supported by UbiCom:

• Context of the physical environment: refers to a dimension of the physical world or phenomena such as location, weather, temperature, precipitation, light intensity, etc.

• Human context (or user context or person context): interaction is usefully restricted by users: in •

identity concepts; preferences; task requirements; social context and other activities; User experience and prior knowledge and types of users.17 ICT context or virtual environment context: A particular component in a distributed system is aware of the services that are internal and external, local and remote, available in the distributed system.

In general, the context-aware focus of UbiCom systems is on awareness of the physical world, often in terms of user models and tasks (Section 5.6). Ubiquitous computers can use their location and physical situation or context to optimize their services in the interest of users. This is sometimes referred to generically as context awareness, but more specifically refers to awareness of the physical context. Greater awareness of the immediate physical environment could reduce energy and other costs associated with accessing physical resources and make systems more environmentally friendly. Consider using your digital camera in your personal visual reminders app. You can know your location and time so that you can record where and when a recording is made. Instead of simply expressing the location in terms of a set of coordinates, you can also use a geographic information system to associate those significant physical objects to that location. It can also know your location, allowing you to print to the closest computer you can reach.

1.2.4.2 User Awareness A camera can be user aware in a number of ways to recognize people and ensure they are in focus, allowing it to configure itself according to a person's preferences and interests. These are all specific examples of awareness of the physical context. User context awareness, also known as persona awareness, refers to ubiquitous services, resources, and devices that are used to support user-centric tasks and goals. For example, a photographer may be primarily interested in capturing digital memories of people (the target of user activity) rather than capturing memories of places or people in places. Because of this, a UbiCom camera can be automatically set to recognize faces and focus on people when taking photos. In addition, in such a scenario, the people in the images can be automatically recognized and given names and named human relationships. Note that the user context awareness property of a UbiCom system, ie h User Context Awareness that intersects with the iHCI property. User context awareness represents a specific subtype of context awareness. A context-aware system can be physically aware

sixteen

UbiCom systems can also have an internal system context, as a system reflects its own internal system operation. The internal context can influence adaptation to the external context. 17 Not only users fully determine the context of a system, but also other stakeholders such as vendors and intermediaries.

Modeling of the most important ubiquitous computing properties

quince

global context, p. the location and temperature of the environment, and knowledge of the virtual world or ICT context, e.g. B. the network bandwidth used for communication (Section 7.6). In practice, many devices today have little idea of their physical context, such as their location and environment. The physical context may not be determined precisely or at all, e.g. B. The camera uses a specific location system that does not work indoors. User context is even more difficult to determine, as user targets may not be published and are often poorly defined. Because of this, the user context is often derived from the user's actions, which in turn can also be ambiguous and non-deterministic.

1.2.4.3 Active versus passive context awareness A key design issue for context-aware systems is balancing the user's level of control and awareness of their environment (Section 7.2). On the one hand, in a (purely) active context-aware system, the UbiCom system recognizes the surrounding context on behalf of the user and automatically adapts the system to the context without the user being aware of it. This can be useful in applications where there are strict time constraints and the user would not otherwise be able to adapt to the context fast enough. An example of this is a vehicle-integrated collision avoidance system that automatically brakes when it detects an obstacle ahead. In contrast, in a (purely) passive context-aware system, the UbiCom system knows the environmental context of the user. It just reports the current context to the user without any customization, e.g. B. A positioning system reports the location of a moving object on a map. A passive context-aware system can also be configured to report deviations from a pre-planned context path, e.g. B. Deviations from a previously planned transportation route to a destination. Design issues include how much control or privacy a human subject has over its context, in terms of whether the subject knows: if its context is acquired, where the context is stored, and to whom and what the context is distributed. Context awareness is discussed in detail in Chapter 7.

1.2.5 Autonomy Autonomy describes ownership of a system that allows it to control its own actions independently. An autonomous system can still interact with other systems and environments. However, he controls his own actions. Autonomous systems are defined as systems that manage themselves and are capable of making their own independent decisions and actions. Autonomous systems can be goal-oriented or policy-oriented: they function primarily to comply with a policy or achieve an objective. There are different types of autonomous systems. On the Internet, an autonomous system is a system controlled by a router policy for one or more networks, controlled by a common network administrator on behalf of a single administrative entity. A software agent system is often characterized as a stand-alone system. Autonomous systems can be designed in such a way that these goals can be dynamically assigned to them, for example by users. So instead of users having to interact and control every low-level task interaction, users only have to interact to specify high-level tasks or goals. The system itself then automatically schedules and schedules the required set of low-level tasks, reducing complexity for the user. The system can also reschedule if a specific plan or task schedule cannot be achieved to achieve goals. Note that the programming problem is often solved using artificial intelligence (AI).

Ubiquitous Computing: Fundamentals and Vision

sixteen

1.2.5.1 Reduced human interaction Much of the ubiquitous system interaction cannot be fully human-centric, even when interaction with computers becomes less intrusive, because:

• Human interaction can quickly become a bottleneck in the operation of a complex system. System scan • • •

be designed so that the human being is in the control circuit. The bottleneck can occur at any step when the user needs to validate or understand that step of the task. It may not be possible to make some or many machine interactions understandable to some people in certain situations. This can overload humans' cognitive and haptic (tactile) abilities, in part due to the sheer number of decisions and the amount of information that emerges. This original vision should be revised and expanded to cover networks of devices that can interact intelligently, to the benefit of humans, but without human intervention. These types of systems are called automated systems.

1.2.5.2 Simplified system maintenance compared to self-managed systems Building, maintaining, and linking individual systems into larger, more open, heterogeneous, and complex systems is more challenging.18 Some systems can be linked to each other relatively easily at the layer network. However, this does not mean that these can be easily linked at the service level, e.g. B. The linking of two independent heterogeneous data sources, which are defined through different data schemas, so that data from both can be aggregated. Such maintenance requires a lot of additional design to develop mapping and mediation data models. The interaction of complex systems, even for automated systems, reintroduces humans to manage and maintain the system. Instead of designing systems that focus on pure automation but ultimately require manual intervention, systems should be designed to operate in a more autonomous manner to work in a self-managed manner to achieve operational objectives. Autonomous systems are related to both context-aware systems and intelligence as follows. System autonomy can be enhanced when a system can determine the state of its environment, when it can create and maintain an intelligent behavior model of its environment and of itself, and when it can adapt its actions to that model and context. For example, a printer can estimate the expected time before the printer runs out of toner based on current usage patterns and notify someone to replace the toner. Note that autonomous behavior does not necessarily always have to be what human users expect and understand, e.g. for example, automatic updating can cause some services to stop responding while these management processes are taking place. Users may need further explanation and mediated support due to perceived differences between the system image (how the system actually works) and the user's mental model of the system (how users understand the system works, see Section 5.5.5 ). From the perspective of software engineering systems, autonomous systems resemble functionally independent systems, where systems are designed as goal-oriented, autonomous functional systems with high cohesion19 and relatively independent from other systems (loose coupling).

18 Operating system software alone can contain more than 30 million lines of code and requires 4,000 programmers to develop (Horn, 1999). The operating system is only one part of the software for a complex distributed system. There are also a number of operating system utilities to consider. 19 Cohesion is the ability of multiple systems or system components to behave as one with respect to specific functions.

Modeling of the most important ubiquitous computing properties

(Presman, 1997). Such systems are easier to design to support composition, defined as atomic modules that can be combined into larger, more complex composite modules. The design of autonomous systems is partially covered in Chapter 10.

1.2.6 Intelligence It is possible for UbiCom systems to be context-aware and autonomous, and for systems to adapt their behavior in dynamic environments in significant ways without the use of artificial intelligence in the system. Systems could simply use a directory service and simple event condition action rules to identify and select among available resources, e.g. B. to discover local resources like the nearest print shop. There are several ways to characterize intelligent systems (Chapter 8). Intelligence can enable systems to be more proactive and dynamic to support the following behaviors in UbiCom systems:

• Modeling of its physical environment: An intelligent system (IS) can adapt its behavior to the actions •

•

most effectively by allowing a model to consider how its environment is changing when deciding how to act. Modeling and imitation of its human environment: It is useful for an IS to have a model of a human being to better support iHCI. IS could allow people to delegate high-level goals to the system instead of interacting with it by specifying the low-level tasks required to achieve the goal. Dealing with incompleteness: Systems can also be incomplete because environments are open to change and because system components can fail. AI planning can support replanning to present alternative plans. A part of the system may be only partially observable. Incomplete knowledge of a system's environment can be supplemented by AI-like thinking about modeling its environment to infer what it cannot see. Dealing with non-deterministic behavior: UbiCom systems can be operated in open and dynamic service environments. User actions and goals may not be fully determined. System design may need to assume that your environment is a semi-deterministic environment (also known as a volatile system environment) and plan how to handle it. Intelligent systems use explicit models to deal with uncertainty. Semantic and knowledge-based behaviour: UbiCom systems are also likely to operate in open and heterogeneous environments. Intelligent system types define powerful models to support interoperability between heterogeneous systems and their components, e.g. B. Semantic-based interaction.

Intelligence types can be divided into single versus multiple intelligence characteristics of entities (see Table 1.5).

1.2.7 Taxonomy of UbiCom properties There are many different examples of the definition and classification of ubiquitous computing. Weiser (1991) referred to UbiCom by function in terms of distribution, discrete access, and context awareness. The UbiCom concept is related to and overlaps with many other concepts, such as Pervasive Computing, Sensitive Computing, Context-Aware Computing, Augmented Reality, and Ambient Intelligence. Sentient Computing is considered a type of UbiCom that uses sensors to perceive and react to its environment. Chen and Kotz (2000) consider that the use of contextual awareness applies more specifically to mobile computing, where applications can recognize and use contextual information (eg, user location, time of day, nearby people and devices). and user activity). Context-aware computing, like intelligent computing, is similar to agent-based computing.

Ubiquitous Computing: Fundamentals and Vision

Computing, in which agents create and maintain a model of their environment in order to operate more efficiently in it. Ambient Intelligence (ISTAG, 2003) characterizes systems in terms of supporting intelligence properties using Ambience and iHCI. Aarts and Roovers (2003) define the five key characteristics of ambient intelligence as integrated, context-aware, personalized, adaptive, and predictive. Buxton (1995) considers ubiquity and transparency to be the two main properties of UbiCom. Aarts and Roovers (2003) classify ubiquitous systems into disposable devices (low power, low bandwidth, embedded devices), cell phones (carried by people, medium bandwidth), and static devices (larger stationary devices with wired connections). Endres et al (2005) classify three types of UbiCom systems: (1) distributed mobile systems, (2) intelligent systems (but their focus here is on embedded sensors and systems rather than intelligence per se) and (3 ) augmented reality Milner (2006) ) considers the three key characteristics of UbiCom as follows: (1) they can make decisions without humans being aware, that is, they are autonomous systems and support iHCI; (2) as systems increase in size and complexity, systems need their personalize services, and (3) more complex unplanned interactions will emerge from the interactions between simple independent components, that is, functions are classified into five main types. or groups of system properties to support the five main requirements of ubiquitous computing (see Figure 1.2). These groups of properties are not exclusive. Some of these subtypes may appear in various types of groups. Here are some examples. Affective or emotive computing can be viewed as subtypes of IHCI and as subtypes of human intelligence. There is often a strong notion of autonomy associated with intelligence, as well as a more distributed system notion. Goal-oriented systems can be viewed as design for intelligence and design for iHCI. Orchestrated and Choreographed can be seen as a way to assemble distributed services and support collective rational intelligence. Custom can be viewed as a context-aware subtype and a subtype of iHCI. Different ideas and visions for ubiquitous computing overlap. There are often different compositions of simpler property types. Ambient intelligence, for example, combines embedded autonomous computing systems, iHCI, and intelligent systems of a social nature. Asynchronous communication allows spatial and temporal separation of components in distributed systems, but it also allows automated systems to do more than just react to incoming events. to interact at any time support. Some properties are similar but are denoted by different terms. The terms pervasive computing and ambient computing are considered synonymous with the term ubiquitous computing. The systems are available to anyone, anywhere, anytime, wherever and whenever they are needed. UbiCom does not purport to imply that all resources, devices, and users in the physical world are ubiquitous, anywhere, anytime, and available to everyone, whether needed or not. Ubiquity, to be useful, often depends on the context, i. h local ubiquity or ubiquity tied to the application domain. The taxonomy proposed in this text is defined at three levels of granularity. At the top level, five main properties are proposed for UbiCom systems. Each of these core properties is defined in terms of more than 70 sub-properties identified in Tables 1.1-1.5. These tables describe more detailed properties of UbiCom systems and the like.20 This allows one type of distributed UbiCom to be defined in the network and mobile sense. Some of these defined sub-properties are themselves such rich concepts that they can be considered sub-sub-properties themselves. For example, communication networks (Chapter 11) include sub-properties such as wired or wireless, service-oriented or network-oriented, etc. Mobility (Chapter 4) can be defined

Without formal definitions of terms at this stage, it is not possible to say that the terms are equivalent and synonymous.

Modeling of the most important ubiquitous computing properties

Table 1.1 Distributed system characteristics Distributed system, middleware, generic service set Universal, seamless, heterogeneous Networked Synchronized, open coordinated Transparent, virtual Mobile, nomadic

Able to work in different homogeneous environments, seamlessly integrating devices and environments, adopting new contexts as new resources become available (Sections 3.2, 3.3). coordinates synchronously or asynchronously through time and space interactions (Section 3.3.3.2) New components can be introduced and invoked, old ones can be modified or removed. Components can be discovered dynamically (Section 3.3.2) Reduces the operational complexity of computing as it acts as a single virtual system, yet is physically distributed (Section 3.4.1) Users, services, data, code and devices can be mobile ( Sections 4.2, 11.7.5)

Table 1.2 iHCI system characteristics Implicit human-device interaction (iHCI) Non-intrusive, hidden, invisible and silent computing Tangible, natural

Predictive, Speculative, Proactive Affective, Emotional Conscious User Posthuman Immersed Sense of Presence Mediated Virtual Reality

ICTs are non-intrusive and invisible to the user. It is integrated into the general ecology of the home or workplace and can be used intuitively by users (Section 5.7). Interaction occurs through natural user interfaces and interaction with physical artifacts, which may involve gestures, touch, voice control, gaze control, etc. (Section 5.3) Improve performance and user experience through expected actions and user goals related to current context, previous user context, and group context. This overlaps with user context awareness (Sections 5.6 and 7.2) Computing, which relates to, arises from, or affects human emotions. This is also considered a subtype of human intelligence (Section 5.7.4 Consciousness) The sense of being in a world that exists outside of us, extending a person's normal experience across space and time (Section 5.4.1) A person is in a real world A time-interactive environment that experiences a greater sense of presence that blends the virtual and the real, often overlaying virtual views with real views (Section 5.4.4)

in relation to sub-sub-sub-features of mobile services, mobile code and mobile hardware resources and devices and in relation to being accompanied, portable and implanted or embedded in mobile hosts. More than 20 different sub-sub-properties are described for autonomous and self-star computing (Section 10.4). These property groups serve to provide a higher level of abstraction of properties important to the analysis and design of ubiquitous systems. It is anticipated that generic distributed system services, such as directory and security services, will also be needed and may need to be designed and adapted to support ubiquitous computing.

Ubiquitous Computing: Fundamentals and Vision

Table 1.3 Context-Aware Sentient System Properties Context-Aware, Unique, Localized, Localized

Systems can recognize and exploit situation or context, such as B. Location, time, and user activity. There are three main subtypes of context awareness; Perception of the physical world, the user and the virtual device (ICT) Systems actively adapt to context changes in a dynamic environment, rather than simply presenting context changes to the user (Section 7.2.4). Details or characteristics provided or collected about a user. This may trigger a system adjustment (Sections 5.6, 5.7). Contextual awareness of the physical world is sometimes understood by some researchers as general contextual awareness or general environmental awareness. Awareness of the physical context includes spatial and temporal awareness (Sections 7.4 and 7.5) Awareness of the ICT infrastructure on which a UbiCom system resides, e.g. B. Knowledge of QoS of the network when transmitting messages (Section 7.6)

Adaptive, Active Context Aware People Aware, User Aware, Personalized, Adaptive, Environment Aware, Context Aware, Physical Context Aware

ICT awareness

Table 1.4 Characteristics of the autonomous system Autonomous Automatic Embedded Encapsulated Embedded Limited resources

Unbound, Amorphous Autonomous, Autonomous, Aspiring Self-Protagonist, Self-Organizer

Operates without human intervention (Section 10.2.1.1) The input-output and computation of the system are fully encapsulated or contained within the device that controls it, e.g. B. a system that functions as a stand-alone application (Section 6.5) sized to be portable or embeddable; limited computing, data storage, input and output, and power to use (Section 13.5.2) Able to work independently and proactively, free of external authority, external dependencies are minimized (Sections 2.2.3.2, 6.4.4) Capable of supporting various auto-star properties such as auto-configure, auto-heal, auto-optimize and auto-protect behavior (Section 10.4) More complex behavior can arise from various simple behaviors (Section 10.5)

Table 1.5 Intelligent system properties Individual intelligent systems Reactive, reflective1

Environment events are captured. Events then trigger action selection, which can lead to actors changing their environment (Section 8.3.2). There are many types of model representations such as rule-based, different types of logic/reasoning from logic-based, etc. Planned and goal-oriented user goals can be used to plan actions dynamically, rather than proactively pre-programmed actions (Section 8.3.4) Utility-based game systems can be designed to work with multiple simultaneous goals (Section 8.3. 5). In theory, adaptive learning systems can be designed to improve their own performance (Section 8.3.6).

Modeling of the most important ubiquitous computing properties

Table 1.5 (continued) Individual intelligent systems Multiple intelligent systems, collective or social intelligence Cooperative Multiple agents can share tasks and information to achieve common collaborative goals (Section 9.2.3) Benevolent Competitive, independent agents and organizations have private goals and profit interests Functions that seek to achieve in a multi-entity environment without requiring antagonistic collaboration, the entities could also act maliciously (Section 9.2.4) by acting as a central programmer; allowing some freedom of interaction mediated by the participants (choreographed) or constrained by the use of a common (mediated) entity or resource (sections 3.3.2, 9.2.2) or a well-defined area of shared meaning (section 8.4) shared knowledge on based on speech acts,2 Multiple agents interact based on propositional attitudes, i. h intentional relationships, based on mentalistic beliefs, desires, or desires and intentions3 (Section 9.3.3.4). Emergent organizations lead to levels of interaction that are not the level of individual interactions (Section 9.2.3.3) 1 Note that a reflex system is different from a reflective system, while the first type of system is designed to respond to environmental stimuli, the last type of system is designed to think about what it is doing (Section 10.3). 2 A speech act-based system differs from a language-based system: while the former uses a particular linguistic theory to form sentence-like structures, the latter is a system that can process human language input and/or convert its output in human language. as discourse 3 Although mental and intentional computation appears as a form of human intelligence, it is generally implemented in the form of a rational model such as BDI logic (Chapter 7).

Each individual property has its own range of a finer set of discrete values, rather than being seen as having or not having a property. Here are some examples:

• • • • • • • • •

from wireless to wired, from ad-hoc to fixed, and from client-server to P2P communication; from full local access only to partial remote access to full remote access; from asynchronous, to synchronous, to coordinated, to organizational, to conventions; from mobility to: static at the point of manufacture; transferred to the place of installation, e.g. B. embedded, then static; mobile between sessions but static during sessions; mobile (roaming from home) during meetings; for homeless free roaming, unbound; from transportable to portable to wearable to implants; from macro to micro to nano; from fully embedded to internally embedded, surface mount, and various forms of loose attachments, such as amorphous computing; from full physical reality to augmented virtual reality, mediated reality, pure virtual reality, and synthetic reality; from working as individuals to working as part of social groups to global interaction.

Section 13.2.2 proposes a multilateral model that offers different levels of support for ubiquitous computing properties, from minimal to full support. Finally, some comments on the terminology and the meaning of system properties and concepts. Different areas of computing may use the same term differently. For example,

Ubiquitous Computing: Fundamentals and Vision

When HCI refers to the adaptive system, it means that the focus of the customization is on the front end of the system or the user interface that adapts to human behavior, and the customization is driven entirely by external concerns. In artificial intelligence, an adaptive system often refers to a system that incorporates machine learning to allow the system to improve or adjust its performance over time. There are also many nuances and different contexts of use of the terms. The grouping of terms in the left column of Tables 1.1 to 1.5 shows that these terms have a high degree of overlap and similarity. This does not necessarily mean that the terms within a grouping are completely equivalent to each other.

1.3 Interaction of the ubiquitous system environment At a high level of abstraction, we can distinguish three types of system environments21 for any particular UbiCom system: (1) other UbiCom systems that form the ICT infrastructure, support services and act as middleware for these applications of ICT systems in particular (virtual worlds);22 (2) human individuals and human organizations; and (3) systems of the physical world,23 including ecological and biological systems. Together, the human virtual (computer) environment and the physical world can be considered as an external environment for UbiCom systems. Note that each of these three main environments seems to have quite different design models and processes. The phenomena of the physical world are governed by well-known laws of physical forces such as electromagnetism and gravity. Living things in the physical world are governed by ecological models of their habitat. Human beings are often governed by economic and social models. A UbiCom system is often conventionally organized as a layered information system stack with a lower layer of information resources, a middle layer of processing, and an upper layer of user information abstractions for displaying and interacting with the information. These can be distributed in different ICT systems through a common communication line. The people who own and operate UbiCom systems and are in the physical world see the physical world and ICT devices as their human environment. Human beings perceive and interact with their environment, often through the visual and tactile senses. Your actions may be driven by a world model that guides your actions and is made up of previous experiences that are learned. The physical environment can be represented with multiple models. In a local physical control model, e.g. B. Lighting controls detect the presence of daylight and turn on artificial light when daylight falls below a certain threshold. More sophisticated control systems may use feedback control. A second type of physical world model is an ecological system, a self-sustaining system inhabited by multiple autonomous organisms that self-regulate their behavior in the face of different pressures and driving events in the system. There are three basic types of environments for UbiCom systems: (1) the infrastructure of other ICT systems; (2) the physical world environment; and (3) the human environment. Several basic types of system environment interactions occur: between humans and ICT systems, HCI (see Figure 1.3); between ICT systems and the physical world, CPI (see Figure 1.4); between ICT systems, C2C or CCI. In addition, interactions can occur between non-ICT systems, e.g. B. between different entities in the physical world and between people (H2H or HHI), also known as social interaction. All these types of interactions coexist. The correlation and simplification of these interactions is

These three types of environments are also known collectively as physical space, cyberspace, and mental space, or the world of atoms, bits, and thoughts. 22 ICT system infrastructure is also called virtual environment or cyberspace. 23 Everything that exists in physical space, natural or artificial, inherently occupies physical space and consumes physical resources, p. For example, a desktop computer often sits on a hard physical surface and consumes power generated by other physical resources in the world. The nature of the physical world in our model is an external environment for a UbiCom system that the UbiCom system can detect and control to support a particular application or usage.

Ubiquitous interaction between the system and the environment

Increased human interaction (H) Human interaction mediated by UbiComp

H2H

H2C/eHCI

Increased physical world interaction (P)

Human beings use the ICT model to interact explicitly. H custom C

C2H/iHCl min.

C is aware of H's context and adapts to it. C customizes C2C

Autonomous interaction with ICT systems

Rise of ubiquitous computing (C)

Minimum

Interaction Figure 1.3 Human-ICT device interaction (HCI) is divided into four interaction subtypes: H2H, H2C, C2H and C2C

Increased physical world interaction (P)

P2P

Increased human interaction (H)

Physical interaction (without ICT mediation) C Acknowledges P C P Context aware P2C/CA C Augments or mediates the reality of P C Adapts to the context of P Context C2P/AR/MR Virtual reality enabled by C

Minimum

C2C/RV

Minimum

Increase interaction with ubiquitous computing (C).

Figure 1.4 The interaction between the ICT device and the physical world (CPI) is divided into four subtypes of interaction: P2P, P2C, C2P and C2C

Ubiquitous Computing: Fundamentals and Vision

discussed further in Section 9.1. Of these three environments, humans have the highest general intelligence and can act most autonomously. This must be taken into account. Human beings are an embodiment of parts of the physical world, but they also cause most of the changes in the physical world. ICT devices are made and work in the physical world. ICT devices have a profound impact on people and lead to changes in social values and norms.24 Each of the HCI and CPI interaction types is illustrated by describing four levels of interaction that span their domain of interaction. This division of interactions into human-physical, physical-computer and human-computer is a framework for analyzing the range of UbiCom systems. Some interactions may span and combine interactions, e.g. B. A mediated reality interaction combines a human-physical interaction and a human-computer interaction.

1.3.1 Human-ICT Device Interaction (HCI) For the interaction between two systems, e.g. B. Humans (H) and ICT systems (C), four characteristic points are considered on the interaction domain, maximum H interaction (H2H or HHI) with minimum C interaction, plus H interaction facilitated by C interaction (H2C), plus interaction C leading to human interaction (C2H) and maximum ICT interaction (C2C). While the humans in H2C have a model of computers, Hz. B. a mental model of C's tasks and goals to interact more effectively with C's use. With C2H, C also has a partial model of H to reduce H's interaction (Section 1.3.3). The interaction between the ICT device and the physical world and the interaction between the human being and the physical world are described in a similar way. ICT Device to Device ICT (C2C), also known as distributed computing systems, is the main focus of computing, telecommunications and networking, but here we take a much broader perspective. C2C facilitates all other types of interactions below, e.g. B. H2H is often mediated by C2C. C2C is often used for task automation and pre-processing to filter out unnecessary resources or filter out needed resources transparently to the user. The ICC, in turn, depending on how it is defined, is intertwined and requires human interaction. Human interaction is necessary in different parts of the life cycle of a CCI system. Humans are involved in the design phase of the UbiCom system, often performing an inspection phase during operation, and are involved in the maintenance phase when design changes are required to keep the system running (Horn, 2001). Kindberg and Fox (2002) state that, in the near term, UbiCom systems will involve humans and that system designers need to make clear the system boundary25 between the ICT system and humans and clarify the responsibilities of both, i. h what ICT systems cannot do and what people will do. People use multiple devices that are explicitly customized by the user and reside in a person's personal and social space.26 People explicitly access interactive and non-interactive multimedia information services for entertainment and leisure. People explicitly access companies (companies) -

For example, in the 1990s, if two strangers were walking down the street and one started talking and waved, one would think the other was talking to him and would respond. In the 2000s, in this case, there is a higher chance that the person is talking remotely to someone else on a cell phone, so the other person is more likely to remain silent. 25 Kindberg and Fox (2002) refer to this ICT-system-human boundary as the semantic Rubicon, named after the Rubicon River in ancient Italy, which marked the boundary of a part of Italy where Julius Caesar was reluctant to use it to his troops. Italy to transcend. 26 See Hall (1996), personal space is the region that surrounds each person, or that area that a person considers their domain or territory. It is fixed up to about 120 cm. This personal space travels with us as we move. These spaces are fluid and multifunctional. The personal space also overlaps with the social space (used for interaction with acquaintances, up to a size of about 360 cm) and with the public space. The exact dimensions and uses of these rooms vary, e.g. B. depending on culture and age.

Ubiquitous interaction between the system and the environment

information related outside the office and various other virtual ICT services to support education, personal productivity, etc. (Explicit interaction between humans and ICT devices H2C or eHCI). ICT applications can use a model of the person that can be created and maintained based on observed user interactions and user activities (implicit interaction between ICT devices and humans, C2H or iHCI). The C of H model can be used to notify users of timely activities, automatically filter and adapt information and tasks, and anticipate and adapt to human actions and interactions. Social and organizational interactions can be mediated through ICT devices (Human to Human, Social, Interaction, H2H). Two people can interact one on one, e.g. B. Unicast voice calls between two people. Computers can facilitate basic information and task sharing, but they can also be used to facilitate a fuller exchange of language, knowledge, experience, and emotions. People can interact within social spaces and within corporate organizational spaces, e.g. B. to support workflows within or between organizations and to collaborate or work together to obtain resources, e.g. B. interact within the auctions to acquire goods.

1.3.2 ICT device interaction with the physical world (CPI) Physical world interaction with the physical world (P2P) refers to interactions within nature that are not (yet) mediated by a major ICT system. There are a variety of simple interactions between animal life used in nature, as opposed to the more complex human-to-human interactions. These include common chemical odors, visual signaling, and various types of audio signals such as drums, buzzes, and voice calls. While this type of biological interaction seems quite esoteric, models of this interaction can be mimicked in CCI and be surprisingly effective in solving some interaction problems. Furthermore, the ways in which organisms interact in their natural habitat to maintain a balanced ecosystem are quite effective models for the self-regulation of autonomous systems (Chapter 10). The interaction between the physical environment and the computing device (P2C) includes context-aware ICT systems. These can be designed to become aware of changes in certain phenomena in the physical world and react to them in a simple way, e.g. B. If the temperature is too low, turn up the heat. ICT systems can also be designed to act on the physical world and change the state of a part of the physical world in accordance with human goals. The interaction of the computing device with the physical environment (C2P) refers to augmented and mediated reality systems. ICT systems are used to augment, supplement, e.g. B. Views of the physical world can be annotated with virtual markers. In the more general cases of mediated reality, reality can be reduced and filtered, not just amplified. The interaction between the physical world and the reality of the virtual world is a strong theme in electronic games. The term hyperreality is used to characterize the inability of human consciousness to distinguish reality from fantasy, a level of consciousness that some electronic game players can reach. In pure virtual reality interaction, computer-to-computer interaction (CCI), the physical world can be used as a conceptual space for virtual interaction. In a virtual reality ICT system, people can wear sensory interfaces, such as gloves and glasses, to allow more natural interaction. Humans may also contain disease implants that can transmit digital data streams to ICT monitoring medical services. Human beings can be virtually represented as avatars to explore and interact with virtual ICT worlds more comprehensively. It is also seen that even with maximum CCI, there is minimal physical world interaction because computers use physical world resources, e.g. Like power consumption. CCI is affected by phenomena in the physical world, wireline and wireless signals are attenuated to different degrees, depending in part on their frequency. In addition, there is a growing awareness that as part of the ecology of the physical world, UbiCom systems work in harmony with it. This must be done throughout the entire life cycle of the UbiCom system, including operation (by optimizing energy consumption) and destruction (through remanufacturing and recycling).

Ubiquitous Computing: Fundamentals and Vision

1.4 Architectural Design for UbiCom Systems: Intelligent DEI Model Three basic architectural design patterns for ubiquitous ICT systems are proposed: intelligent devices, intelligent environment27 and intelligent interaction (Figure 1.5). Here the term intelligent simply means that the entity is active, digital, connected, can operate autonomously to some extent, is reconfigurable, and has local control over the resources it needs, such as power, data storage, etc. It follows that these The three main types of system design contain subsystems and components at a lower level of granularity that can also be considered intelligent, e.g. B. An intelligent environmental device may consist of intelligent sensors and an intelligent controller, etc. There's even smart dust (Section 2.2.3.2). Figure 1.5 provides an illustrative example of how these three types of models can be used. These are many examples of intelligent subtypes28 for each of the three basic types of intelligence, which are discussed in detail in later chapters of this book. The three main types of intelligent design also overlap, they are not mutually exclusive. Smart devices can also support smart interaction. Intelligent mobile launch devices can be used for control in addition to using static embedded environmental devices. Smart devices can be used to support virtual views of smart personal spaces (physical environment) in a personal space that accompanies the user everywhere. Satyanarayanan (2001) has also postulated different architectures and ways for the development of UbiCom systems, firstly, to develop from distributed systems to mobile distributed systems to ubiquitous computing, and secondly, to develop UbiCom systems from intelligent spaces.

Device Trends Increasing ability to manufacture more complex, low-power micro-devices

Increase the ability to integrate devices into the physical environment

Increased capacity for more interoperable distributed mobile devices

Use more complex, multifunctional, mobile, personalized (and private) smart devices to make services easier to access and embed rather than just virtualize, e.g. B. Is a phone also a camera, a music player or a printer?

Use smarter environments to detect and respond to events such as people with mobile devices entering and leaving controlled spaces, e.g. B. The walls can detect that the camera is recording and change the lighting to improve the recording

Use more service access devices with simpler functions and let them work together: smarter interaction between devices, e.g. B. The camera can be connected to the phone to share recordings or directly to the printer to print

Ubiquitous computing Figure 1.5 Three different models of ubiquitous computing: intelligent terminal, intelligent interaction and intelligent infrastructure

27 Note: Some people think of the smart environment model simply as ubiquitous computing, but ubiquitous computing is also referred to here as a smart device model, e.g. B. Mobile communication devices and intelligent interaction model considered. 28 More levels of intelligence subtype granularity could be added, p. B. Subtypes of intelligent embedded environmental devices, such as implants, but these are not provided to simplify Figure 1.6.

Architectural design for UbiCom systems: Smart DEI model

Smart Devices Smart DEI Model

intelligent interaction

Smart Environments

knowledge

ASOS

VM-MTOS

real time operating system

mobile service

wireless

Rules

Sensor

intelligent agent

Organization

multiagente

label data

boards

task pad

Eyelash

Staub

robot

self* cooperative

competitive single agent

MEMS

nanotechnology

Figure 1.6 Some of the main subtypes (triangular relationships) of intelligent devices, environments and interactions and some of their main aggregations (diamond relationships), where MTOS is a multitasking operating system, VM is a virtual machine, ASOS is an application. Specific or embedded operating system, RTOS is a real-time operating system and MEMS is a microelectromechanical system

Invisibility, localized scalability and unequal conditioning. The Smart DEI model is similar to Satyanarayanan's, except that it also includes smart interaction. Smart DEI also refers to hybrid models that combine smart devices, smart environments, and smart interaction designs (Figure 1.6). Gillette et al. (2000) speculate that general-purpose end-user devices will persist but evolve to a more modular form, driven by user frustration with the proliferation of devices with overlapping capabilities and a desire for consistency across multiple environments (e.g., at home, in the car and at the office). . This is a motivation for smart interaction and more for the smart device model. In practice, however, users seem to be very tolerant of the vast majority of devices with overlapping features that are inconsistent and often not interoperable.

1.4.1 Smart devices Smart devices, e.g. B. PCs and mobile phones are often multipurpose ICT devices that act as a single portal to access popular multi-application service sets, which may be local to the device or remote to servers. There are a number of forms for smart devices. Smart devices are typically personal devices with a specific owner or user. In the smart device model, the control site and the user interface reside on the smart device. The main characteristics of smart devices are the following: mobility, dynamic discovery of services and intermittent access to resources (concurrency, updating, etc.). Devices are often designed to be multifunctional, as they make it easy to access multiple functions at runtime and simplify their interoperability. However, the trade-off is less system openness for hardware component maintenance (upgrading) and support for more dynamic and flexible runtime interoperability.

Ubiquitous Computing: Fundamentals and Vision

1.4.1.1 Weiser Forms of ICT Devices: Tabs, Pads, and Tablets We often think of computers primarily as personal or server computers with multiple applications, as devices with some kind of display for data output and a keyboard and some kind of pointing device for data. entrance. As humans, we routinely interact with many more devices that contain single embedded computers, such as household appliances, and with complex machines29 that contain multiple embedded computers. Weiser found that there was a trend from many people per computer30 to one computer per person31 to many computers per person. Computer-based devices tend to get smaller, lighter, and cheaper to manufacture. Therefore, devices can prevail, become more portable and appear less intrusive. Weiser considered a variety of device sizes in his early work, from centimeter-sized wearables (tabs) to decimeter-sized wearables (pads) to meter-sized displays (tablet). ICT pads for people to access mobile services and ICT tabs for tracking goods are widely used. Wall displays are useful for multi-person viewing, collaborative work, and viewing large, complex structures such as maps. The on-board units can also be used horizontally as a surface computer, as well as in a vertical position.

1.4.1.2 Advanced shapes for ICT devices: dust, skin and clay The three shapes proposed by Weiser (1991) for devices, tabs, pads and boards are characterized by: macro size, flat shape and by the incorporation of visual output spectacles. If we relax each of these three properties, we can expand this realm to a much more diverse and potentially useful realm of ubiquitous computing devices. Firstly, ICT devices can be miniaturized without visual output screens, e.g. B. Microelectromechanical systems (MEMS), ranging from nanometers to micrometers and millimeters (Section 6.4). This form is called Smart Dust. Some of them can combine multiple tiny mechanical and electronic components, allowing an increasing number of functions to be integrated into ICT devices, the physical environment and humans. Today, MEMS such as accelerometers are embedded in many devices, such as laptops, to detect falls and station moving components, such as disk arms, and are increasingly being integrated into widely deployed systems. They are also used in many devices to support gesture-based interaction. Miniaturization, accompanied by cheap manufacturing, is a key enabler for the vision of ubiquitous computing (Section 6.4). Second, light-emitting and conductive polymer-based fabrics, organic computing devices, can be molded into more flexible non-flat display surfaces and products such as clothing and drapery (Section 5.3.4.3). MEMS devices can also be painted on various surfaces, allowing a variety of structures in the physical world to function as the meshed surfaces of MEMS (Section 6.4.4). This form is called Smart Skins. Third, MEMS arrays can be formed into arbitrary three-dimensional shapes as artifacts that resemble many different types of physical objects (Section 6.4.4). This form is called Smart Clay.

For example, new cars have several dozen built-in computers and sensors to help with brake assist, airbag inflation, etc. to about a billion people it means. 31 The phase from a computer to a person may not have been around for very long, depending on how one defines a computer. Certainly the people who had personal computers back then also had a lot of embedded digital devices.

Architectural design for UbiCom systems: Smart DEI model

1.4.1.3 Mobility Mobile devices generally refer to communicators, multimedia entertainment, and business processing devices intended to be carried by their human owners, e.g. B. Mobile phones, game consoles, etc. There are different types of mobile phones:

• Accompanied: These are devices that are not carried or implanted. They can be portable or handheld, separate but worn inside clothing or fashion accessories.

• Laptop: Such as laptops designed for two-handed seated use. These are generally the devices with the highest resources.

• Handheld – Devices are generally operated with one hand and occasionally hands-free, where:

•

numerous applications such as communication, audio and video recording and playback, and mobile office. These are low-resource devices. Wearable: Devices such as accessories and jewelry are usually operated hands-free and work autonomously, e.g. B. Watches that act as personal information managers, headphones that act as audio transceivers, glasses that act as visual transceivers, and contact lenses. These are low-resource devices (Sections 2.2.4.5, 5.4.3). Implanted or Embedded – These are often used for medical reasons to augment human functions, e.g. B. a pacemaker. They can also be used to improve the abilities of people with physical and mental abilities. The implants can be silicon-based or carbon-based macro- or micro-sized integrated circuits, e.g. B. Nanotechnology (Section 6.4).

Static can be seen as an antonym of Mobile. Static devices are typically moved to a fixed location prior to installation and then remain there for their entire operational life cycle. They typically use a continuous network connection (wired or wireless) and a fixed power supply. They may contain a high level of local computing resources, e.g. B. Personal computers, AV recorders and players, various household and office appliances, etc. The separation between static and mobile devices can be more detailed. For example, static could be moved between usage sessions, e.g. B. a mobile circus with various rides instead of the rides of a fixed amusement park. Mobile ICTs are covered in detail in Chapter 4.

1.4.1.4 Access to ephemeral services Mobile phones tend to use wireless networks. However, mobile phones can sometimes be connected to wireless networks (WAN is not always available) or wired communication networks (LAN to LAN migration) or both. Access to the Service by smart mobile devices is characterized as follows. Intermittent devices (service access) access software and hardware services intermittently. This may be because the resources are finite and the demand exceeds the supply, e.g. B. when a device runs out of power and you have to wait to recharge it. This may be because the resources are not constantly accessible. Service discovery: Devices can dynamically discover available services or even changes in the service context. Devices can detect the availability of local access networks and connect to remote home network services through core networks. You can discover local resources and offset the cost and availability of local access to remote services. Devices can be designed to access the intermittent services they discover. Contextual discovery can improve basic discovery by limiting discovery to the services of interest instead of alerting you to many services that don't match the context. With asymmetric remote service access, there tends to be more downloads than uploads. This is partly due to limited local resources. For example, due to the higher power required to transmit rather than receive communications and the limited power capacity, high power consumption results in more

Ubiquitous Computing: Fundamentals and Vision

calls received than sent. Apart from the ability to generate and transmit voice signals, early telephones were designed as transceivers and players. Recently, due to miniaturization, mobile devices not only function as media players, they can also function as media recorders and content sources.

1.4.1.5 Localized and Aware Smart devices, while capable of remotely accessing any Internet service, tend to use different contexts to filter information and access services. For example, the devices can be operated to focus on local views of the physical environment and maps, and access local services such as restaurants and hotels. Mobile devices are often designed to work with a reference location in the physical environment called a home location, e.g. B. Mobile network nodes report their temporary location addresses to a home server, which is used to help coordinate mobility. Service providers often charge for access to mobile service access services based on their distance from an ICT reference location, an ICT origin location. During transit, cell phones tend to refer to a route from a place of departure to a place of destination. Mobile devices support limited local hardware, physical, and software resources in terms of performance, display, CPU, memory, etc. They are limited by ICT resources. Services accessed or pressured to use such devices must take these limitations into account, otherwise the use of resources by the services will be sub-optimal and may be wasted, e.g. B. by receiving content in a format that cannot be played. In the latter case, the mobile device could act as an intermediary to send that content to another device where it can be played. Unlike an external power supply, mobile devices tend to use finite internal power storage, which increases mobility. The internal power supply can be provided by a natural, self-renewing external source, e.g. B. Solar energy, or supplemented with an artificial energy network: energy self-sufficiency. This is especially important for low-maintenance wireless devices. Devices can automatically configure themselves to support different features based on available power. Without an internal power cache, the mobility of devices can be limited by the length of the power cable to which they are connected. There is usually a one-to-one relationship between mobile phones and their owners. The configuration and operation of the devices tends to be customized to support the concept of a personal space of information and services that accompanies people everywhere.

1.4.2 Smart Environments In a smart environment, computing is seamlessly used to enhance ordinary activities (Coen, 1998). Cook and Das (2007) define an intelligent environment as “an environment capable of acquiring and applying knowledge about the environment and its inhabitants to improve their experience of that environment”. A smart environment is a set of connected devices that have some connection to the physical world. Unlike smart devices, the devices that make up a smart environment typically perform a single, predefined task, e.g. B. motion or body heat sensors coupled with a door opening and closing control. Components of the embedded environment can be designed to automatically respond to or anticipate user interactions using iHCI (implicit human-computer interaction), e.g. B. when a person approaches a closed door, so that the door opens automatically. Therefore, smart environments support a limited local context of user interaction. Smart environment devices can also be fixed in place or mobile in the physical world, e.g. B. in the air. Smart environments might require the addition of new and revolutionary updates to the environment to support less intrusive interaction, e.g. For example, pressure sensors can be built into surfaces to detect when people are sitting or walking. A more evolutionary approach could add minimal modifications to the environment by incorporating devices such as surface-mount wireless sensors, cameras, and microphones.

Architectural design for UbiCom systems: Smart DEI model

1.4.2.1 Marking, Capturing, and Controlling Environments Intelligent environment devices support multiple ways to interact with environments such as the physical environment (Chapter 6) as follows:

• Labeling and annotation of the physical environment: labels, eg. B. RFID32 tags can be attached

•

• • •

•

physical objects Tag readers can be used to find the location of tags and track them. Virtual labels can be attached to virtual views of the environment, e.g. for example, a label can be attached to a location on a virtual map. Sensing or monitoring of the physical environment: Transducers take inputs from the physical environment to convert some phenomena in the physical world into electrical signals that can be digitized, e.g. B. how much ink is in the cartridges of a printer. Sensors provide raw data about the state of the physical environment as input to determine context in a context-aware system. Detection is often a precursor to filtering and tuning. Filtering: A system forms an abstract or virtual view of a part of its environment, such as B. the physical world. This reduces the number of features in the view and allows viewers to focus on the features of interest. Adapt: The behavior of the system can adapt to features of interest in the environment or adapt to changes in the environment, e.g. B. A physical proximity route is based on the relationship of the current location to a destination. domain of the physical world. Controllers generally need sensors to determine the state of physical phenomena, e.g. B. Heating or cooling systems that measure the temperature in an environment. The control can include actions to modify the state of the environment to cause it to go to a different state. Control may involve changing the order (arrangement) of artifacts in the environment or regulating the physical environment. Assembly: Robots are used to act on a part of the physical world. There are a variety of robots. They may be pre-programmed to plan a series of actions in the world to achieve a specific goal, such as For example, a robot may integrate sensors to detect objects at a source location, move them, and stack them at a destination location (palletizing). The robots can be stationary, e.g. B. a robotic or mobile arm. Regular: Regulators usually work at a fixed location, e.g. B. a heating system uses feedback control to regulate the temperature in a room within a selected range.

1.4.2.2 Integrated versus unconnected smart environments contain components that have varying degrees of dependency on their physical and ICT environments. Intelligent environments can use embedded or unconnected components. Embedded devices are static embedded in a larger system, which can be static or mobile. Embedded systems generally provide control and sensing support for a larger system. Devices may be embedded in: (1) parts of physical environments, e.g. B. An access system controlled by passengers or vehicles; (2) Parts of the human environment, e.g. B. cardiac pacemaker; and (3) parts of larger ICT devices, e.g. for example, a tracking device might be built into a phone or camera instead of externally attached. Unbound, amorphous, or spray devices are types of environmental devices that can mix with other particles and spread on surfaces or disperse in gases and liquids, e.g. B. Smart Dust (Sections 2.2.3.2, 6.4.4). They are nomadic or untethered devices that do not need to be operated at a site of origin (base). They can be organized to optimize their operation (Section 10.5.1).

RFID tags are also often referred to as smart tags or smart tags, but smart tags in this text encompass a much broader range of tags and more specific properties (Section 6.2).

Ubiquitous Computing: Fundamentals and Vision

1.4.2.3 Device sizes Smart Environment devices can vary in size. This affects your mobility. Macro-sized devices span a variety of device sizes, from tab-sized devices (size of a centimeter) to block-sized devices (size of a decimeter) and plate-sized devices (size of a meter) (Section 1.2.2.2). Microelectromechanical systems (MEMS) are manufactured using embedded chip technology. This enables large-scale (thousands to millions) inexpensive fabrication of integrated circuit-like devices that can be distributed on the surface or transported by air. Nanotechnology are devices ranging in size from 1 to 100 nanometers that are made up of molecular components. These are built from larger molecules and materials that are not controlled at the atomic level (more feasible) or are chemically assembled according to molecular recognition principles (less feasible).

1.4.3 Smart Interaction For smart devices and smart environments to support the core features of UbiCom, an additional type of design is required to tie together its many individual activity interactions. Intelligent interaction is required to promote a unified and continuous interaction model between UbiCom applications and its UbiCom infrastructure, the physical world and human environments. In the intelligent interaction design model, system components are dynamically organized and interact to achieve common goals. This organization may be internal without external influence, be a self-organizing system, or may be partly driven by external events. Components interact to accomplish goals together because they are not deliberately designed to perform and complete bundles of goal-achievement tasks on their own; they are not monolithic components of the system. Designs based on sets of interacting components have several advantages. There are a variety of levels of interaction between the components of the UbiCom system, from the simple to the clever. A distinction is made between (basic) interaction, which uses fixed interaction protocols between two statically connected dependent parties, and (intelligent) interaction, which uses richer interaction protocols between multiple parties or dynamic independent entities.

1.4.3.1 Basic interaction A basic interaction normally involves two dependent parties: a sender and a receiver. The sender knows in advance the recipient's address; the structure and meaning of the exchanged messages are agreed in advance, the sequencing, ie the order of the individual messages, is known in advance. However, the content, the instances of the message that adhere to the accepted structure, and the meaning may vary. There are two main types of basic interaction, synchronous or asynchronous (Section 3.3.3):

• Synchronous interaction: The interaction protocol consists of a control flow of two messages, a

•

then ask for a reply or an answer. The sender sends a request message to the specified recipient and expects to receive a response,33 p. B. A client component makes a request to a server component and receives a response. Asynchronous interaction: The interaction protocol consists of individual messages that have no flow control, a sender sends a message to a receiver without necessarily knowing if the recipients will receive the message or if there will be a subsequent response, e.g. B. an error message is generated, but it is not clear if the error is handled and results in a response message.

33 This quotes synchronization at the interaction level, the sender is waiting for a response. Synchronization can also be done at the message level instead of the interaction level. If the flow control of each message is synchronized, the sender waits for an acknowledgment received by the receiver.

Architectural design for UbiCom systems: Smart DEI model

1.4.3.2 Intelligent Interaction Asynchronous and synchronous interaction are considered as part of the communication functions of distributed systems (Section 3.3.3.2). In contrast, coordinated, convention-based, semantic, and language-based interactions whose interactions are driven by dynamic organizations are considered intelligent interactions (Section 9.2.3). Therefore, smart interaction extends the basic interactions as follows:

• Coordinated interactions: various components work together to achieve a common goal

•

explicit communication, e.g. B. A sender requests a receiver to process a request to complete a subtask on behalf of the sender, and the interaction is synchronized to accomplish this. There are different types of coordination, such as B. Orchestration (using a central coordinator) versus choreography (using a distributed coordinator). Policy and convention-based interaction: Different components work together to achieve a common organizational goal, but are based on agreed rules or contractual guidelines without necessarily requiring significant explicit communication protocols between them. This is based on previously understood rules to define norms and abnormal behaviors and the use of commitments by members of organizations to adhere to guidelines or norms, z each other and move with the center of gravity, etc. Dynamic organizational interaction: Organizations are systems that require an arrangement of relationships (interactions) between individuals in such a way that they create a system with properties that do not exist at the level of individuals. Extensive types of mediation can be used to task others in organizations with completing tasks. There are many types of organizational interaction protocols, such as auctions, brokers, contract networks, subscriptions, etc. Semantic and Linguistic Interactions: Communication, interoperability (common definitions of how communication is used), and coordination are enhanced when the components involved share common meanings. of the interchanged terms and share a common language to express basic structures for the interchanged semantic terms.

Consider a scenario where lightweight resources are designed to be context-aware to conserve power. They are designed to be actuated by human presence. When they detect the presence of a human, they automatically turn on. If they do not detect any presence, turn off to save energy. However, if there were multiple lights in a dark room and they were only context sensitive, they would all turn on when someone walked in, wasting power unnecessarily. If they instead supported intelligent interaction, they could decide among themselves which lights to turn on to best support specific human activities and goals. Smart interaction requires devices to interact to share descriptions of resources (eg desk lamp, wall lamp, main ceiling light) and goals (eg read, watch a video, retrieve something). This example is more complex in practice, as you may need to support multiple users and potentially conflicting user goals. Intelligent interaction also requires an intelligent orchestrating unit (central planner) or a choreographer unit (distributed planning) to set goals and be able to plan tasks with the participation of others to achieve those goals. Resources and users could compete with each other and participate in markets where a utility value is assigned to the use of a resource and users must submit the best offer to purchase the use of a resource (auction interaction). Resources can interact and organize to provide a combined service (Chapter 9).

Ubiquitous Computing: Fundamentals and Vision

1.5 Discussion 1.5.1 Linking System Properties, Environments, and Designs Table 1.6 (and Figure 1.7) provides a further comparison of Smart DEI models in relation to internal system and system environment models. There are several ways to model how smart devices, smart environments, and smart interactions interconnect. Each of these intelligent designs are related to the three main types of environments, human, physical and virtual. For example, intelligent environment designs focus on the distribution of multiple devices in the physical environment that are aware of their human users and their physical location. Smart environmental devices tend to operate more autonomously compared to smart devices and require minimal human operational management. Smart devices may also support some

Table 1.6 Comparison of smart device, smart environment and smart interaction type

smart device

smart environment

intelligent interaction

Characteristics

Active multifunction devices based on a virtual computing environment

Interaction with the system environment.

Weak CPI, strong H2C, weaker C2H and strong C2C

Active single function devices integrated or dispersed in a physical environment Strong C2P and C2H

Individual components that must work together or compete to achieve their goals Rich H2H, P2P models applicable to HCI and CPI

Dynamic ICT service, resource discovery

Dynamic discovery of physical resources

Dynamic composition of entities and services

medium low

Hoch

medium low

High interaction, personalized, 1.11

Shared 1-M interaction between users

High Some innovative iHCI in smart ICT devices

Medium Very different user interfaces. iHCI focuses on context awareness Located in part(s) of the physical world Autonomous control of local ICT resources

M-M Interaction Coordinate and organize interactions based on Low Language and iHCI

Distributed system: openness, dynamic services, mobility of ephemeral access to ICT Context awareness: the physical world, users and ICT infrastructure

HCI: Place of Control

Autonomy located in the ICT device

intelligence

Autonomous control of local ICT resources, less autonomous control of remote services Low to medium individual rational intelligence

Low to medium individual rational intelligence

Distributed in the physical and virtual world High autonomy of actions and interactions

High collective intelligence: semantic sharing, social cooperation and competition

1 1.1, 1-M, and M-M refer to the cardinality of the interaction and represent one-to-one, one-to-many, and many-to-many interactions, respectively.

discussion

DEI smart model

smart device

1-1 interaction

smart mobile device

1-M, M-M interaction

intelligent interaction

smart environment

Mobile, custom, MTOS, 1-1 remote interaction

fixed vs. No strings attached, ASOS, Sense-Control, local 1-1-Interaktion

Design: Fat-Clientproxy-Server?

Design: Autonomous interaction between peers?

Diseño Thin-Client-Proxy-Server-Interaktion?

1-M, M-M, richer (cooperative, competitive, semantic, etc.), P2P interaction design: application model, service set, remote service access points, service contract, on-demand, autonomous?

Figure 1.7 Alternative views and organizations for the device, environment, and units of interaction in the Smart DEI model

Awareness of the physical environment as spatial awareness. Intelligent interaction uses the stronger notion of intelligence in terms of intelligent selection, mediation, and composition when multiple cooperating and competing entities interact compared to other intelligent designs. An important division34 of the Smart DEI model is the separation of the interaction between individual users and UbiCom systems, modeled as part of the smart device design, and the interaction between groups, such as social and economic groups of users, modeled as part of the model is an intelligent interaction design modeling. The social and economic models of people can also be used for the interaction between multiple ICT devices and services in virtual environments.

1.5.2 Common myths about ubiquitous computing Ubiquitous computing is a pretty broad view. There is a risk that it becomes too wide. Discussed here are some unrealistic expectations of ubiquitous computing:

• There is only one definition that aptly characterizes ubiquitous computing, rather there are a number of properties and types of ubiquitous computing that vary by application.

This division may seem artificial, but it serves to separate design concerns. Although many-to-many interactions naturally occur between services, devices, and human users, this is often modeled as a one-to-one interaction using mediators that serialize the interactions. In practice, individual interactions naturally overlap or need to be combined with group interactions, e.g. B. Using a phone in a meeting. The design of smart devices and smart environments can also be combined with smart interaction design depending on the application.

Ubiquitous Computing: Fundamentals and Vision

• The ideal type of ubiquitous computing is when all the properties of ubiquity must be fully satisfied •

•

• •

Supported: It may not be necessary, useful, or practically usable in many cases to support the full set of these properties. Ubiquitous computing means making computing services accessible everywhere: it is unnecessary, too expensive, and leads to smart environments becoming too cluttered, overloading the user with too many options, and going against the idea of hidden computing. Ubiquitous computing is also about data processing being located within a context and only available when needed. Therefore, it is more appropriate to speak of context-aware ubiquity. Ubiquitous computing is unlimited computing: this means that the virtual world of ICT can be fully extended to the physical world and the human environment, replacing human and physical world systems and their interactions with computer interactions. However, the performance of computer systems is limited, at least in the short term, e.g. For example, UbiCom systems are not (yet) capable of fully displacing human cognition and behavior. As such, UbiCom must strike a careful balance between supporting humans and living in harmony with and experiencing the physical world, between designing to give people more satisfying control over their environment and taking away less satisfying control over the environment. Ubiquitous computing is all about HCI: automated interaction and decisions are also needed to reduce human cognitive and task overload and perform tasks more safely, quickly, repeatably, and accurately. It is also less convenient for people to interact with micro-sized devices in the same way as macro-sized devices. Human interaction with compounds of multiple devices that are spatially distributed in a shared physical space and time cannot be controlled centrally in the same way that humans can control a single device. Calm Computing should be used as a model for all HCIs. Quiet computing is where the system is active, reducing some human decision-making processes. There are many applications and situations where human users need to clearly direct and control the interaction. Quiet computing should be used in a targeted way. From weak to strong, degrees of calm arithmetic are required. Ubiquitous computing is all about augmented reality: UbiCom systems can not only enhance the interaction between humans and the physical world, but also change it in more comprehensive ways. It can even diminish reality in some way to help the user focus on specific contexts. UbiCom is more about reality than media. Ubiquitous computing is nothing more than distributed or virtual computing: UbiCom is more than being distributed in the sense of networked, transparent and open ICT systems. UbiCom also focuses on specific models of human-physical world interaction that involve contextual awareness of the human and physical world, and support implicit human-computer interaction. Ubiquitous computing is nothing more than mobile wireless computing: the ability to carry more powerful multifunctional wireless mobile devices is useful but also limited. Too many features can lead to clutter. More and more functions can interfere with each other. Making mobile devices strongly local and adapting to the physical world can be complex. Ubiquitous computing also affects location and integration in the physical world. Ubiquitous computing is all about smart environments: While smarter interactions with the physical world can be facilitated by incorporating active computing into the real world, UbiCom is also embracing smart, flexible, and self-centered mobile device interactions. human and allows spaces for personal and social interaction. Ubiquitous computing must be a highly autonomous system: the autonomy of systems is often limited in practice because computers cannot design themselves, fully adapt to new environments and user requirements, and keep up with changing changing requirements. Ubiquitous computing is all about context awareness of the physical world: many types of context-aware systems are episodic, considering only the current state of the physical environment to determine their next actions. This is partially observable and not effective.

discussion

•

nondeterministic world. Furthermore, the context of the physical world must be seen as part of the user's context. Ubiquitous computing is just distributed intelligence: stock selection and many trades can be too complex and computationally difficult, requiring significant computation to allow intelligent reasoning to arrive at a result. The interaction is more efficient and easier to calculate and execute when it has minimal intelligence, e.g. B. in the design of reactive systems, rule-based behavior as used in self-organizing and self-generating systems. However, intelligence is very useful when systems need to deal with uncertainties and handle autonomous systems that are themselves complex and intelligent. Ubiquitous computing systems can function effectively in all kinds of environments: It is unrealistic to expect that ubiquitous computing systems can behave deterministically in non-deterministic, partially observable, etc. human and physical environments. Today's systems cannot reliably and actively adapt to user contexts in which users act ad hoc. A weather context aware system cannot reliably and accurately predict what clothing users should wear when the weather itself is unpredictable.

1.5.3 Organization of the Smart DEI approach The chapters of the book are structured as follows:

• Foundations: Foundations and Vision (Chapter 1); Applications and Requirements (Chapter 2); • Smart Devices: Smart Devices and Services (Section 3); Smart cell phones, cards and device networks (Chapter 4); Human-Computer Interaction (Chapter 5);

• Intelligent Environments: Labeling, Detection and Control (Chapter 6); Context Aware Systems (Chapter 7);

• Intelligent Interaction: Intelligent Systems (IS) (Chapter 8); Intelligent System Interaction (Chapter 9); Autonomous Systems and Artificial Life (Chapter 10).

• Middleware and Outlook: Pervasive Communication (Chapter 11); Smart device management (Chapter 12); Prospects and challenges of the generalized system (Chapter 13). This book can be studied in various ways. The traditional way, starting with the basics and then reading to understand more advanced topics that build on that. This tells a story from more abstract and underlying concepts to more technology-driven approaches. UbiCom is multidisciplinary with some common themes across all disciplines, including: Foundations and Vision (Chapter 1); Applications and Requirements (Chapter 2), Pervasive System Challenges and Outlook (Chapter 13) (see Table 1.7).

Table 1.7 Book chapters and their relationship with the Smart DEI (Smart Device, Environment, Interaction) model, the properties of the UbiCom system and the interaction between system and environment No

chapter title

SINCE

Property of UbiCom

environmental interactions

1 2

Basic and vision applications and requirements

GODS GODS

all all

Smart devices and services

Devices

All distributed, iHCI, distributed according to context

C2C

(continues overleaf)

Ubiquitous Computing: Fundamentals and Vision

Table 1.7 (continued) No

4 5 6 7 8 9 10 11 12 13

chapter title

SINCE

Property of UbiCom

environmental interactions

Smart mobile devices, device networks and smart cards Human-computer interaction Labeling, detection and control Context awareness Smart systems Smart interaction Autonomous systems and artificial life Communication networks Smart device management

Devices

Distributed

C2C

Devices Environment Environment Interaction Interaction Interaction

iHCI Context Aware Context Aware Intelligent Intelligent, iHCI Autonomy, Intelligence, Distributed Distributed, iHCI, Context Aware All

HCI IPC IPC C2C, HCI H2H, C2C C2C

Challenges and perspectives of the generalized system

Devices DEI Devices

C2C C2C, HCI, CPI, todos

These disciplines in the book can be studied separately. This book is a good way to illustrate its practical application. These subject-specific topics are studied in addition to the basic part (chapters 1 and 2) as follows:

• UbiCom: all chapters, but especially HCI (chapter 5), tagging, detection, physical world control (chapter 6), context awareness (chapter 7).

• ICT: Smart devices and services (Chapter 3), smart mobile devices, device networks and smart cards (Chapter 4), ubiquitous communications (Chapter 11), smart device management (Chapter 12).

• HCI: HCI (Chapter 5), Context Awareness (Chapter 6), Smart Device Management in User-Centric Environments (Section 12.3), Pervasive Systems Challenges and Perspectives (Sections 13.6, 13.7, 13.8).

• AI: Intelligent Systems (Chapter 8), Intelligent Interaction (Chapter 9). Autonomous systems and artificial life (Chapter 10).

EXERCISES 1. Suggest some new ways to improve the use of personal visual memory presented in section 1.1.1. (Hint: What if the camera was portable? What if the camera supported other forms of imaging, such as text recognition?) 2. Discuss why there is no paperless information environment in the world, but why what seems to be happening in a world of photography without film. Compare and contrast the use of paper and photos with computer notepads as information systems in relation to their support for UbiCom requirements. 3. Give a few more examples of current ubiquitous computing applications and suggest future ones. 4. Analyze and distinguish between three different definitions of ubiquitous computing. 5. Discuss the benefits of ubiquitous systems: supporting a strong notion of autonomy, supporting a strong notion of intelligence.

references

TRAINING

(Continuation)

6. Discuss whether ubiquitous systems must fully support all five ubiquitous system properties and whether or not the five ubiquitous system properties are independent or strongly related. 7. Discuss the need for intelligent UbiCom systems, argue for and against. 8. Discuss the point that conventional computing focuses on computer-to-computer interaction, while ubiquitous computing also largely focuses on the interaction between the ICT device and the physical world and the interaction between the ICT device and the physical world. the human being 9. Describe the range of interactions between humans and computers, computers and the physical world, and humans and the physical world. Illustrate your answer with specific applications of the UbiCom system. (Additional exercises are available on the book's website.)

References Aarts, E. and Roovers, R. (2003) Ambient Intelligence: Challenges in Embedded System Design. In Proceedings of Design Automation and Test in Europe, Munich, pp. 2-7. Buxton, W. (1995) Pervasive media and the active office. Published (only) in Japanese as Buxton, W. (1995). Widespread Video, Nikkei Electronics, 3.27(632), 187-195. English translation retrieved from http://www.billbuxton.com/UbiCom.html, April 2007. Chen, G. and Kotz, D. (2000) A Survey of Context-Aware Mobile Computing Research. Technical Report TR2000-381. Available at http://citeseer.ist.psu.edu/chen00survey.html. Accessed November 2006. Coen, M.H. (1998) Design principles for intelligent environments. In Proceedings of the 15th/10th National Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence, pp. 547-554. Koch, DJ and Das, S.K. (2007) How smart are our environments? An updated look at the state of the art. Pervasive and Mobile Computing, 3(2): 53–73. Endres, C., Butz, A. & MacWilliams, A. (2005) A review of software infrastructures and frameworks for ubiquitous computing. Mobile Information Systems, 1(1): 41-80. Gillett SE, Lehr W, Wroclawski J and Clark D (2000) A taxonomy of Internet devices. TPRC 2000, Alexandria, VA. Retrieved from http://ebusiness.mit.edu on 2006-09. Greenfield A (2006) Everyware: The Dawning Age of Ubiquitous Computing. London: Pearson Education. Hayles, N. K. (1999) How we became posthumans: virtual bodies in cybernetics, literature, and computing. Chicago: University of Chicago Press. Horn, P. (2001) Autonomic Computing: IBM's Perspective on the State of Information Technology, also known as IBM's Autonomic Computing Manifesto. Retrieved from http://www.research.ibm.com/autonomic/manifesto/autonomic_computing.pdf, accessed November 2007. ISTAG. (2003) IST Advisory Group, Advisory Group for the Information Society Technology Program of the European Community. Ambient intelligence: from vision to reality. Retrieved from http://www.cordis.lu/ist/istag.htm. May 2005 article. Kindberg, T. and Fox, A. (2002) System Software for Ubiquitous Computing. IEEE Generalized Computing, 1(1):70-81. Milner, R. (2006) Ubiquitous computing: should we understand it? The Computer Journal, 49(4):383-389. Pressman, RS (1997) Design for real-time systems. In Software Engineering: A Professional's Approach. Maidenhead, 4th Edition: McGraw-Hill, pp. Rogers, Y. (2006) Advancing Weiser's vision of Calm Computing: Engaging UbiComp Experiences. In P. Dourish and A. Friday (eds.) Proceedings of Ubicomp 2006, Lecture Notes on Computing Science, 4206: 404-421.

Ubiquitous Computing: Fundamentals and Vision

Russell, S. and Norvig, P. (2003) Artificial intelligence: a modern approach. 2nd Edition Englewood Cliffs, NJ: Prentice Hall. Satyanarayanan, M. (2001) Pervasive Computing: Vision and Challenges. IEEE Personal Communications, 8:10-17. Schmidt, A. (2000) Implicit human-computer interaction through context. Personal Technologies, 4 (2 & 3): 191–199. Warneke, B., Last, M., Liebowitz, B., and Pister, K.S.J. (2001) Smart Dust: Communicating with a cubic millimeter. Computer Science, 34(1):44-51. Weiser, M. (1991) The computer for the twenty-first century. Scientific American, 265(3): 94-104.

2 Applications and requirements 2.1 Introduction Ubiquitous computing posits a world in which people are surrounded by computing devices and a computing infrastructure that supports us in everything we do. People will live, work and play in an integrated computerized environment nested in the world. The focus is on the development of ubiquitous computing systems to help people in their daily activities in the physical world, making them simpler and less intrusive. Bushnell (1996) coined variations of the term commodity, such as B. Desk, sofa, kitchen, car, bedroom, and bathroom items1 to reflect the use of ubiquitous computing for routine tasks. More recently, Greenfield (2006) revised this idea and coined the term everyware to encompass the different types of goods.

2.1.1 Overview Many ICT companies and research institutes have implemented UbiCom initiatives. First, this chapter provides an overview of a selection of notable early projects (Section 2.2). The survey excludes intelligent and autonomous system projects, as these are discussed in detail later in this book. Second, UbiCom's current ICT infrastructure and Everyware UbiCom suites are discussed (Section 2.3). Third, an analysis of these projects is carried out and the main design challenges are highlighted (Section 2.4).

2.2 Examples of early UbiCom research projects Below are some notable early research on UbiCom projects. These are also considered in terms of their current technological (ICT) context.

1 See http://wearcam.org/safebath_leonardo/safebath_leonardo207s.txt.htm for an article on smart bathroom accessories and systems by Steve Mann, accessed February 2008. 2 There are many innovative UbiCom projects that impose space constraints on this section. only a selection of which can be reproduced here. Two of the main conferences, covering a broader range of UbiCom projects, can be found on the IEEE website www.UbiCom.org and on the ACM website www.percom.org.

Stefan Poslad

Applications and Requirements

2.2.1 Smart devices: CCI According to Weiser et al. (1999) Work on ubiquitous computing began in 1987 at the Xerox Palo Alto Research Center, or PARC3, when Bob Sprague, Richard Bruce, and others proposed developing wall-sized displays. At the same time, PARC anthropologists, led by Lucy Suchman, looked at actual work practices and PARC's leading technology researchers to reflect on how computers were embedded in the complex social framework of daily activities. Weiser's early work dealt with the proliferation of computers that used wireless communication in novel ways. Her subsequent work focused more on iHCI. Out of these converging forces, from atoms to culture, emerged the UbiCom program at PARC in 1988.

2.2.1.1 Smart Boards, Pads and Tabs UbiCom's first program proposed three mainly interlocking devices and applications: a large-wall display program called LiveBoard, which transitioned from amorphous silicon to rear-projection; smaller computers, the book-sized MPad and the palm-sized ParcTab computer. These became known as tabs, pads, and tables. Initial systems were designed to support access and use of distributed information by mobile users (tabs and pads). Keep in mind that in the late 1980s, wireless networks were not ubiquitous or integrated with wired networks. The tabs and pads in particular have been designed so that not only device identification, but also its location, status, usage, connectivity, and owner are contextual. The ParcTab system consisted of palm-sized mobile computers that could communicate wirelessly with workstation-based applications via infrared transceivers. Designed for portability, the ParcTab can also be used or carried around, e.g. B. tied to a belt. It should foster on-demand computing, always-on connectivity, and assisted location awareness. All prototype pads, tabs, and badges were fully functional and used daily by PARC experimenters. At its peak in 1994, some 40 members of the PARC lab used ParcTabs in their daily activities. The goal of LiveBoard was to support collaborative group design and work. Later it became a commercial product.

2.2.1.2 Active badge, club and floor The active badge system of Want et al. (1992) and the associated positioning system was developed in 1989 at Olivetti Research Laboratories/Cambridge University. This was perhaps the first context-aware computing application designed to improve user mobility and aid in location awareness. It was intended to be an aid to a telephone receptionist, before cell phone networks became widespread, so that employees could be contacted when they were away from their desks or at home. Once a person was located, phone calls could be routed to the desk phone closest to the person's location (Section 7.3.1). An active badge periodically sends infrared signals to sensors embedded in rooms throughout the building. This became the precursor that led to the development of ParcTab at PARC when Roy Want left and later joined PARC. The location accuracy of the Active Badge system was limited to determining which room a person was in. In 1995 Ward et al. (1997) developed a new active tag system called Active Bat, which is based on ultrasound4 and can locate people with an accuracy of up to 3 cm. The system used a base station to ask the bat for a signal.

3 4

See http://www.parc.com, accessed June 2007. Hence the name Bat because bats use ultrasound in the wild to determine location.

Example of UbiCom’s first research projects

it is then measured at various ceiling receivers, where position is determined by trilateration (Section 7.4.1). Unlike the active badge and active bat, the active floor or smart floor did not require anyone to explicitly use an identifier. Instead, this type of device is designed to indirectly identify people, e.g. B. in this case by his gait or gait (Addlesee et al., 1997). Active floor design requires careful analysis to specify the proper spatial resolution and robustness to allow users to walk on the sensors without damaging them.

2.2.2 Intelligent Environments: CPI and CCI 2.2.2.1 Classroom 2000 A potentially useful feature of future computing environments will be the ability to capture the live experiences of its occupants and provide users with recordings for later access and review. In 1995, a group at the Georgia Institute of Technology undertook a three-year project called Classroom 2000 to try to support both teaching and learning at a university by introducing automated support for lecture capture (Abowd, 1999). While most of the work in teaching material development has focused on the development of multimedia materials, Classroom 2000 sought to improve content generation by providing a room with the ability to record a lesson5 instead of students manually taking their own notes and use them later, they may need to transcribe them. . The project had two goals: (1) to understand the issues in designing a ubiquitous computing application that provides effective ways to capture and access rich live experiences; and (2) understand what it takes to develop a robust and ubiquitous computing application whose impact can be evaluated in its target domain over a long period of time. The first prototype used a large electronic whiteboard system that allowed the instructor to view and annotate slides. Students can also use a tablet to annotate their own copy of the slides. These were not initially networked. A second classroom prototype, specially equipped for use with Classroom 2000, opened in January 1997. Microphones and video cameras were embedded in the ceiling and signals from the microphones and cameras were recorded. The electronic whiteboard was used again, but this time online. Two ceiling-mounted projectors connected to networked computers were also on display.

2.2.2.2 Smart Space and Meeting Room The NIST Smart Space and Meeting Room projects, 1998-20036 focused on the use of ubiquitous devices, sensors, and networks to provide an infrastructure for context-aware smart meeting rooms. and respond to ongoing human activity. (Standford, 2003). It was divided into two phases. In Phase 1, experimental smart spaces were prototyped, focusing on the following key areas: advanced forms of human-computer interaction, integration of picocellular wireless networks with dynamic service discovery, automatic device configuration, and the software infrastructures required for the successful programming of ubiquitous. Computer Science - Applications are required. Phase 2 focuses on developing metrics, test methods,

5 Instrumentation in this context refers to the process of adding sensors that act as sources of information in the physical world. These sources can be configured and registered for online and offline analysis. Other systems, such as software-based systems, can also be instrumented to monitor their operation and reflect and even modify their behavior. 6 Estimated from published work.

Applications and Requirements

and standard reference data sets to advance the technology and provide reference implementations that serve as models for potential commercial implementations. The meeting room digitized signals from two hundred microphones, five video cameras, that is, direct sensor data, and had a smart whiteboard. Two sets of tools are used to manage this sensory data. Sensor flows are managed by the NIST SmartFlow system, a data flow middleware layer that provides a data transport abstraction and consistent formats for data flows. The metadata or annotations of the data stream are addressed with semantic descriptions using Architecture and Tools for Linguistic Analysis Systems (ATLAS). As people talk to each other in the meeting room, the system can record dictations, record a transcript of the meeting, track individual speakers, track what the conversation is about, and activate related services from the Internet. This supports an iHCI model for taking notes and assisting speakers by intuitively providing more information.

2.2.2.3 Interactive Workspaces and iRoom The Interactive Workspaces project began at Stanford University in 1999 and studied the design and use of interactive spaces called iRooms, which contain one or more large screens, with the possibility of integrating devices portable and create applications that integrate this use of multiple devices in the user space (Johanson et al., 2002). There were several design principles:

• Emphasize shared location: use shared physical space to •

• •

Orientation and interaction, p. B. Team meetings in individual rooms. This is in contrast to teleconferencing and video conferencing, which connect multiple physical rooms together. Relying on social conventions to make systems understandable: The space is designed to give a group the necessary opportunities to adjust the environment as they go about their task, i. h Users and social conventions take responsibility for actions. The system infrastructure is responsible for providing a liquid medium to perform these actions. Broad Applicability (Interoperability): Create standard abstraction and application design methods that apply to any interactive workspace. Keep it simple for the user (understandable): the system should remain accessible for casual users with no experience, and on the software development side, you should try to keep the APIs as simple as possible so that the client-side libraries are easier to use. wear. port as well as the barrier of entry for application developers.

The second version of the iRoom included three whiteboard-sized touch screens on the side wall, a large screen that supported stylus interaction, a so-called interactive wall built into the front wall, and a custom screen designed to resemble a standard screen. conference table The room also had cameras, microphones, WiFi support, and a variety of wireless buttons and other interaction devices. The space supported three main interactive tasks: moving data, moving control, and dynamically coordinating multiple applications.

2.2.2.4 Cooltown HP's Cooltown project, 2000-2003, developed a vision for UbiCom to help mobile users provide access to information through web-based wireless handheld devices and virtual ICTs that connect the world with the physical world (Kindberg et al., 2000). A key feature of the Cooltown approach is that a physical world resource can have a web presence. Physical resources are associated with a simple standard resource identifier, URL, or universal resource locator. URLs of physical and virtual resources can be easily found and exchanged locally

Example of UbiCom’s first research projects

Device-physical world transfers. Thus, when entering a new room, a PDA device can receive a message containing that room's URL via IR transmission from a beacon in the room, by reading a barcode in the room, etc. The PDA can then activate the access. Check the room's website for available facilities and features. Kindberg et al. indicate that using the web for mobile users who are in the physical world offers three key benefits:

• Pervasive Access: The Web offers accessibility that supports mobility in two ways. resources

• •

The web can be accessed from any device that supports the standard HTTP protocol and that supports transparent access to resources anywhere on the web, as long as the ephemeral criterion is supported (see Section 2.2.4.5). Just enough middleware (thin client access): Web standards are used and simple URLs are exchanged. No specific application programming language middleware is required to run on the device, just a web browser. (Section 3.3.3.4). Stay local whenever possible: Web services can be delivered to mobile users via a local web server without the need for a global wireless connection such as a cell phone or mobile IP. This minimizes the amount of infrastructure required for users to interact with local services.

2.2.2.5 EasyLiving and SPOT Microsoft has carried out several UbiCom research projects and developed several product prototypes, this work is ongoing. These include: the EasyLiving project, 1997-2002, the Smart Personal Object Technology (SPOT) initiative, started in 2003, and the MyLifebits project (Section 12.2.9.2). The focus of the EasyLiving project, 1997–2003, was the development of intelligent environments that enabled the dynamic aggregation of various I/O devices into a single, coherent user experience (Brumitt et al., 2003). With UbiCom applications, there is often a need to separate hardware device control, internal computational logic, and user interface presentation, rather than tightly coupling input/output devices to applications. For example, to allow content created on one device to be streamed to another device and controlled by yet another device, or because the input control device is not conveniently co-located with a remote display device, such as on the remote control of a big screen. show EasyLiving enables this type of flexible interaction by providing abstract descriptions of its capabilities, geometric models of the position of devices relative to other devices, and sensor capabilities. By contrast, the desktop PC model assumes that computer peripherals such as the screen, mouse, and keyboard are connected to a single computer and are all in one convenient physical location. When working in a distributed environment, it is no longer possible to assume this static configuration of fixed devices, both in terms of device presence and physical configuration. In the SPOT initiative, SPOT devices were developed to listen to digitally encoded data, such as news, weather forecasts, personal news, traffic information, and retail listings, transmitted on frequency sidebands leased by commercial VHF radio stations (Krumm et al., 2003). ). . Such devices could send valuable broadcast notifications and alerts to millions of people.7

7 The SPOT initiative appears to be stagnant, although there are many potential applications for it. For example, electronic devices containing a timer chip could also be designed as VHF receivers and automatically synchronize their time with a radio transmission, rather than requiring users to manually set the time through a proprietary control interface. A local radio receiver could also act as a store and forward gateway and connect a local cost short range network, e.g. B. Bluetooth, to distribute this data. The previous use of VHF radio as a data channel to support UbiCom is not used enough. Perhaps the increasing availability of digital radio broadcasts will act as an enabler.

Applications and Requirements

2.2.2.6 HomeLab and Ambient Intelligence Ambient Intelligence or AmI was proposed by Philips in the late 1990s as a new paradigm for consumer electronics that is sensitive and responsive to human presence, e.g. B. Knowledge of personality. This was considered to have a slightly different approach than the original ubiquitous computer vision. AmI was developed within Philips into a company-wide strategy called Healthcare Lifestyle Technology (Weber, 2003). In 2002, after two years of planning and construction, the Philips HomeLab was launched, a state-of-the-art laboratory that could be used to conduct feasibility and usability studies in the field of ambient intelligence. Philips HomeLab looked and felt like a normal home with modern furniture in every room and even a fully equipped kitchen. Ambient intelligence prototypes have been installed, ranging from electronic devices that can recognize speech and movement to digital displays on the bathroom mirror and new "toys" designed to help children expand their creativity. Although no one lives at Philips HomeLab, temporary "residents" can stay on the premises for as long as necessary, depending on the type of research being conducted. Researchers can closely monitor how their tenants live with these technologies 24 hours a day through tiny cameras and discreetly hidden microphones in the HomeLab. During the first year of HomeLab, three consumer needs were explored:

• Need to belong and share experiences: research focuses on connectivity as a support enabler

• •

share content and experiences and create a sense of togetherness. For example, the presence of a distant group was transmitted electronically to another group in different ways while they shared a common experience, such as watching a football match (de Ruyter et al., 2005). Need for emotions, excitement and relaxation: Research focused on enhancing experiences by adding intelligence to remotely controlled light and sound environments. Need to balance and organize our lives: Research focused on developing intuitive navigation concepts that put users in control through the development of a context-sensitive Personal Remote Control (PRC). This used context awareness to filter events. Using an array of sensors, the remote could recognize contexts, such as B. being in the user's hand, lying in a drawer, the user nearby, etc. The remote also changed the way it issues reminders for upcoming shows based on personal meaning to users.

In parallel, the vision was exported and developed as part of an EU-wide research framework (ISTAG, 2003). The two key properties of intelligent environmental systems are as follows. Connected devices will be integrated into the environment, including sensor networks and embedded control systems. User-aware systems8 can self-adapt by adapting to changes in the personal context related to a person's goals. The system has a model of a person's goals and behaviors. Anticipates a person's goals and automatically adjusts or adjusts to them. It is this trait of consciousness of the person that primarily implies a specific social type of (environmental) intelligence, in the sense that systems are designed to work closely with their human users.

2.2.3 Smart Devices: CPI 2.2.3.1 Unimate and MH-1 Robots Machines are used to perform physical tasks that are labor intensive and repetitive, or too dangerous or difficult to be performed directly by humans. Automated machines that only do one thing

User awareness combines four overlapping properties described by Aarts (2003): context awareness, personalized, adaptive, and predictive.

Example of UbiCom’s first research projects

they are not robots Robots have the ability to perform a variety of programmable tasks, such as: B. in a factory, possibly using different end effectors, i. h hands or claws. In 1961, Heinrich Ernst of MIT developed the MH-1, a computer-controlled mechanical hand (Ernst, 1961). The first computer-controlled industrial robot, the Unimate designed by Joseph Engelberger, followed an earlier patent he filed in 1956. It had a powerful five-jointed arm. It could be programmed to load and unload machine tools, palletize parts, drive welding guns, and operate die casting machines and forging presses. The first application was in 1962 for die casting at a General Motors automobile factory.

2.2.3.2 Smart Dust and TinyOS Microfabrication and integration of low-cost computer sensors, actuators and controls, MEMS (Micro-Electro-Mechanical Systems) allow devices or particles small enough to be sprayed or airborne to be embedded in a digital system. environment and create a digital skin that perceives a variety of interesting physical and chemical phenomena (Section 6.4.4). This has been termed Smart Dust9 (Warneke et al., 2001). Unlike the Internet, where data is often generic and can become outdated, the information in this digital skin can be located, updated, and directly accessible to end users and applications. The University of California, Berkeley (UCB) Smart Dust project, circa 2001, led by Kris Pister, investigated whether an autonomous sensor, computing and communication system could encapsulate a cubic millimeter particle (a small particle or speck) to form the foundation

Figure 2.1 Example of Smart Dust, Golem Dust, solar power splitter with two-way communication and sensing, acceleration, and ambient light, approximately 10 mm3 total circumscribed volume and 5 mm3 total displaced volume. Reproduced with permission from Warneke, B.A., Pister, K.S.J. (2004) An ultra-low power microcontroller for intelligent networks of wireless dust sensors. International Conference on Solid State Circuits. 2004, (ISSCC 2004): 316-317. OR 2004 IEEE

The name "smart dust" apparently started as a joke, according to Kris Pister (Frost, 2003): "Everyone was talking about smart homes, smart buildings, smart bombs, and I thought it was fun to talk about smart dust."

Applications and Requirements

integrated and massively distributed sensor networks (Figure 2.1). The Smart Dust Mote consisted of either a thick-film battery, a solar cell with a charge-integrating capacitor for dark times, or both. Depending on the application, it integrates different sensors. An integrated circuit provides sensor signal processing, communication, control, data storage, and power management. A photodiode enables optical data communication. TinyOS started in 1999 as a collaboration between UCB and Intel Research (Hill et al., 2000) and has since grown into an international consortium in 2007. TinyOS is an open source embedded operating system and platform for wireless sensor networks. (WSN). applications. It should be integrated into Smart Dust and its follow-up projects.

2.2.4 Smart Devices: iHCI and HPI These projects here are sort of combinations of smart device types, smart environment, and smart interaction that focus on supporting iHCI.

2.2.4.1 Calm Computing In the mid to late 1990s, Weiser became more interested in a vision of UbiCom, which he called Calm Technology or Calm Computing (Weiser and Brown, 1997). Weiser noted that while personal computers and games have been focused on the excitement of interaction, we interact with them differently when computers are everywhere. We often want to calculate while doing something else. The term "periphery" refers to what we are attuned to without explicitly caring about it. This is comparable to foreground processing and attention vs. background processing and attention. Things on the periphery are tuned in by the large part of our brain devoted to peripheral (sensory) processing. Calm Technologies are designed to put us at ease, as they can strengthen our periphery in three ways:

• include both the center of our place of attention and the periphery of our attention moving back and forth between the two;10

• Improve our range of peripherals by adding more detail to the periphery. the periphery informs

•

without overwhelming us. The periphery has a larger storage capacity than the focus, e.g. B. Unlike a conference call, a video conference allows you to focus on facial expressions and postures that are otherwise inaccessible. Offering location awareness: When our periphery is working well, we are in tune with what is happening around us, and therefore with what is about to happen and what has just happened. We are effortlessly connected to a multitude of familiar details.

An example of silent technology was the "Dangling Rope" created by artist Natalie Jeremiyenko. This is an 8 foot long piece of plastic spaghetti that hangs from a small electric motor mounted to the ceiling. The motor is electrically connected to a nearby Ethernet network cable, so any information passed through causes a small jolt in the motor. Therefore, the level of twitching indicates the level of network traffic on that Ethernet segment.

2.2.4.2 Thinking things and tangible bits The Things That Think11 (TTT) research consortium was formed at the MIT Media Lab in 1995 to study how the physical world meets the logical or virtual computing world. can be grouped

10 11

It is not clear how exactly this can be controlled or manipulated. Things That Think, http://ttt.media.mit.edu/, accessed January 2008.

Example of UbiCom’s first research projects

in bits, people and atoms, i. h virtual computing environment, human environment and physical world environment (Gershenfeld, 1999). There are quite a few projects that have been carried out at MIT, such as Oxygen and Tangible Bits (see below). Gershenfeld leads the physics and media group, TTT's Bits layer provides an overview of some of the projects members of his group have been involved in, such as E-Ink (section 5.3.4), laptops (sections 2.2.1.4, 5.4 .3), the use of digital computers to improve stringed musical instruments, body area networks (Section 11.7.4) and electronic money. The MIT Tangible Bits project, led by Ishii (Ishii and Ullmer, 1997), began in 1995 as part of the TTT program. This explored a future where intelligence, sensing, and computing spread from the desktop to "things." Conventional computer systems are largely characterized by exporting a GUI-style interaction metaphor to large and small computer terminals residing in a virtual environment. By contrast, the Tangible Bits project aimed to transform the "painted parts" of a GUI into "tangible parts" by making greater use of multiple senses and multimodal human interactions with the real world, and resembles the Calm vision. Weiser's Computing in his later work. Tangible user interfaces emphasize visually intense hands-on interactions in the foreground and background perceptions of ambient light, sound, airflow, and waterflow at the periphery of our senses. Three specific applications have been developed called metaDESK, transBOARD and AmbientROOM. The metaDESK consists of a graphical surface projected almost horizontally. Physical objects summoned to this surface were detected by an array of integrated optical, mechanical, and electromagnetic field sensors. Users could use these physical objects to interact with views of the virtual world from satellite images (augmented reality). The transBOARD is a digitally enhanced, connected physical whiteboard designed to explore the concept of interactive surfaces that absorb information from the physical world, convert that data into bits, and distribute it in cyberspace. The ambientROOM used ambient media, ambient light, shadows, sound, airflow, waterflow as a means to convey information at the periphery of human perception. For example, the sound of heavy rain indicates many site visits and success in attracting customer attention, while no rain can indicate poor marketing or possible web server failure.

2.2.4.3 DataTiles The focus of the Sony DataTiles project (Rekimoto et al., 2001) was an interactive user interface using task-specific physical objects as an alternative to manipulating virtual information systems, rather than input devices for data. general purpose such as a mouse and keyboard (see figure 2.2). This approach has several potential benefits. Physical objects can offer better deals12 than purely visual and virtual ones. This allows people to use additional haptic skills to manipulate objects, not just point and click, but rotate, grab, snap, etc. Interactions can involve two hands. It allows multiple people to interact cooperatively in the same physical interaction space. Unlike virtual GUI objects, physical objects do not suddenly appear or disappear when the system changes modes, nor do they become unresponsive when the system is busy with other tasks. Another motivation behind this work is the increasing complexity of orchestrating digital devices: users need to be able to focus on the task itself and not on the underlying ICT system. The abundance of devices for specific tasks, information devices, will be the main interfaces with the digital world, interacting with each other to support our daily lives. The DataTiles system consisted of clear acrylic tiles with embedded RFID tags; a flat screen that also serves as a tile tray, an electromagnetic pen tablet behind the screen, RFID readers (sensing coils) mounted behind the glass cover of the screen; an electronic circuit for detection

Things that suggest obvious actions based on shape and other attributes, e.g. B. a knob or a dial can be turned.

Applications and Requirements

Figure 2.2 The DataTiles system integrates the benefits of two main interaction paradigms, graphical and physical user interfaces. DataTiles reproduced with permission from Ó Sony Computer Science Laboratories, Inc.

multiple sensor coils with a single RFID reader (Section 6.2.4). Three key types of interaction have been incorporated into the system. DataTiles can act as physical windows to obtain information and trigger specific actions when the tiles are placed on a sensor-based display surface. Various combinations of physical and graphical interfaces are possible, e.g. B. Grooves can be etched into tile surfaces and also serve as passive haptic guides for pen operations. Of course, a simple physical language is implied to combine multiple tiles, e.g. for example, stacking multiple tiles in a certain order might imply a sequence of actions.

2.2.4.4 Ambient Wood The Ambient Wood project, started in 2002, created a slightly different excursion in which children had to discover, hypothesize and experiment with biological processes taking place in an outdoor physical environment (Rogers et al. , 2004). In addition to being able to explore the environment for themselves, students were able to use tools that digitally augmented the environment and allowed them to make their own measurements of the area. The information of its position triggered a variety of other details about the environment and its inhabitants. A digitally enhanced sonar tool was available to the children, allowing them to collect information about moisture and light in their habitats. The readings could be displayed on a laptop as an image showing relative values instead of numerical ones. Information about the position of the children in the forest could be recorded and relevant information about the location of the creatures in the forest could be transmitted to your handheld.

2.2.4.5 WearComp and WearCam Wearable computing is considered here as a more specific case of surface mounting on a mobile host. When the mobile host is a human, it is called a portable ICT device and not a surface mount device. Surface mounted devices, in turn, are considered a subspecies of mobile ICT devices.

Example of UbiCom’s first research projects

Figure 2.3 Type of wearable computing devices, Mann's prototypes Source: Reproduced with permission from http://en.wikipedia.org/wiki/Wearable_computing, Ó Steve Mann

(Section 4.2). One of the pioneers of portable computing13 is Steve Mann, whose first experiments with portable computers began in the late 1970s. His first major application focused on recording personal visual memories that could be shared with others over the Internet. In an overview of his work, Mann (1997) distinguished between three (overlapping) generations of portable computing (see Figure 2.3). All of these used a head-mounted display permanently available in the user's field of vision to support augmented or mediated reality, a wireless transceiver to exchange information with external units and nodes, and a programmable computer for local processing. The third generation, from the mid-1980s to 1997, enabled computer-assisted forms of interaction in everyday situations, walking, shopping, and meeting. The operation time was measured in days and the body net circuit used clothing threads as conductors. This new generation supported three key features: Hidden computing, e.g. B. Custom glasses have been used as a head mounted display; a special conductive fabric was used as the body area network; Mediated reality and homographic modeling were used. Homographic modeling14 allows annotating the view with text or simple graphics. It was found that by calculating and mapping the aircraft's homographs, an illusory rigid flat patch appeared to float above objects in the real world, resulting in a form of computer-mediated reality, e.g. For example, a person could leave a virtual shopping list at a glance on a refrigerator that would be intended for a specific person. Although the message is sent immediately, it remains in the recipient's WearComp memory until a corresponding homographic view triggers the message. This has the advantage that information can be hidden to reduce user overload until an appropriate context, in this case a visit to the supermarket, triggers the message.15 This represents a form of context-aware stigmergy (Section 10.5.1 ) and prospective memory ( Section 5.7.6). Laptop applications include: monitoring the physiological functions of the human body, the distance traveled, the route taken, and projecting hands-free information for viewing.

13 For the history of wearable computing, see http://www.media.mit.edu/wearables/lizzy/timeline.html, accessed November 2007. However, the definition of computer here is vague and includes some analog devices that have since been replaced by digital versions. The question also arises as to what constitutes a computer. UbiCom includes MTOS computers (Section 3.4.3), embedded computing and microprocessor applications (Section 6.5). 14 Homography is defined as a relationship between two figures such that any given point in one figure corresponds exactly to a point in the other and vice versa. 15 One can easily imagine many uses for it, e.g. B. Someone can leave a note ahead of time for someone to eat healthier, which is only triggered when the webcam sees that something is being eaten.

Applications and Requirements

Figure 2.4 A series of electrodes surgically implanted into Warwick's left arm and connected to median nerve fibers are monitored. Reproduced with permission of the Ó University of Reading

2.2.4.6 Cyborg 1.0 and 2.0 ICT devices implanted in human mobile hosts are considered here as a more general case of embedded devices, i. h Devices embedded in inanimate or animate devices (Section 4.2.1), which in turn are considered a specific type of mobile ICT devices. In 1998, Kevin Warwick16 underwent an operation in which a silicon transponder chip was surgically implanted in his forearm, which remained in place for 9 days (see Figure 2.4). This experiment, Cyborg 1.0, enabled RF readers embedded in the environment to respond to Warwick as he moved through the corridors and offices of the University of Reading Department of Cybernetics (Warwick, 2003). A unique identification signal emitted by the implanted chip could be used to unlock doors, lights, heaters and other computers without him lifting a finger. However, this implant, although inside his body, was not part of his body as it was connected to him. Furthermore, carrying a handheld transponder or a card transponder would trigger the same events in the physical environment. A second experiment, Cyborg 2.0, was launched in 2002 and investigated how a new implant in his forearm could send signals between Warwick's nervous system and a computer (Warwick, 2003). With his arm, he could remotely control an electric wheelchair and an intelligent artificial hand. This required a period of training to condition his brain to recognize signals, and his brain was only able to adapt to those signals for about an hour. He was also able to create artificial sensations by stimulating individual electrodes within the matrix and sharing them with another human, his wife, who had a second, less complex implant connected to his nervous system. The implant was removed after approximately 90 days when the implanted sensor was close to failure.

2.2.5 Other UbiCom projects This text contains many other examples of UbiCom projects17 and smart applications. Mobile devices and apps are discussed in more detail in Chapter 4. additional information

sixteen

See also http://www.kevinwarwick.com, accessed December 22, 2007. Some projects frequently cited in the literature are not included because detailed peer-reviewed scientific publications on their operation could not be retrieved. 17

Everyday applications in the virtual, human and physical world

on HCI projects and applications are covered in Chapter 5. Micro- and nanoscale devices and robotic projects are covered in Chapter 6. Context-aware projects and applications are covered in detail in Chapter 7. Here are also some brief sketches of other smart environment projects. The Aura Project (Garlan et al., 2002) at Carnegie Mellon University aimed to introduce the concept of a personal information aura: an invisible halo of information and computer services that persists regardless of location and propagates through portable devices, portable and desktop devices, extends and infrastructure computers. He examined how to create a home environment that is aware of the whereabouts and activities of its occupants. The Aware Home Research Initiative at the Georgia Institute of Technology, led by Abowd, began in 1998 and is still ongoing. Simulate and evaluate user experiences with standard and cutting-edge technologies applied to the home. It explores how residents can be offered services that improve their quality of life or help them maintain their independence in old age (Kidd et al., 1999). The Cognitive Lever or CLever project, started in the early 2000s to study computing environments, was designed to not only help people with a variety of cognitive disabilities, but also its supportive community dedicated to goal-focused approaches to intelligence augmentation (AI) to complement, strengthen, and extend human capabilities (Kintsch and dePaula, 2002). Other UbiCom projects and applications are also considered in the next section.

2.3 Everyday Applications in the Virtual, Human, and Physical World The UbiCom applications that this book focuses on are the everyday applications where UbiCom systems facilitate people's routine activities. UbiCom's applications can be grouped by the type of interaction they enable: human-computer interaction, human-physical world interaction, and computer-physical world interaction and human-human interaction.

2.3.1 Pervasive Device Networks: CCI's information and communication technologies (ICTs) have become a critical part of the global infrastructure. Three types of wide area ICT networks are now quite ubiquitous: (1) GSM and other wireless telecommunication networks; (2) TCP/IP-based wireless access networks connected to a wired Internet backbone; and (3) satellite networks, including Global Positioning System (GPS) networks. The TCP/IP-based Internet, a network of heterogeneous networks, is increasingly being used as a universal backbone to support many different logical media applications, such as 11). Video streams, voice unicasts, and data can often be bundled as triple services from different content, service, and network providers and delivered over a single wide area network. These can continue to be divided into individual media content and accessed by individual non-interoperable devices. If we add mobile information services to this package, they will be called quad-play services. There are many more service packages that could be considered. If we add radio audio streams, we could call them a Pentad Play service package. Mobile devices such as phones, PDAs and cell phones already offer Pentad Play service packages.18 We can also create other service delivery packages, e.g. B. Sextet set and septet sets to combine audio and video interactive games or power distribution, etc. (see Section 2.3.2.1).

Currently, many other types of devices, such as game consoles and various audio and video streaming receivers, may also offer such pentad streaming services, but are not designed to do so (see Section 2.5).

Applications and Requirements

Within specific domains, such as B. the home domain, TCP/IP-based WLANs and fixed local area networks can be easily configured to distribute multimedia content within this domain. However, it is also common to use a computer as a hub to access application services and to use remote media such as memory cards, CDs and DVDs, and short serial line connections to exchange content. The means for accessing and distributing AV and data content received within a local domain can be easily expanded to allow distribution outside the home. Video content licensed at home could be moved for access outside the home using different media store and forward services. This is called place shifting and contrasts with time shifting, playing media streams to pause or slow them down. Some countries have adopted specific strategies to target ubiquitous services and applications. For example, since 2001, the Nomura Research Institute in Japan has reported a more limited vision of ubiquitous computing for a ubiquitous networked society in the form of the u-Japan plan (Murakami, 2004). Murakami considers two main strategies for ICT development: the development of improved ICT infrastructure and the promotion of better use of ICT. This vision focuses on the evolutionary development of today's mainstream network ICTs rather than more futuristic ICT computing such as digital ink, paper and portable computing, called esoteric computing. The development of the ubiquitous network is considered in three dimensions:

• everywhere: on the PC, in other rooms, outdoors, in moving vehicles. • Anytime: indoors on the PC, indoors away from the PC, while traveling (in another location, eg stores, movies, friends, etc.), and while traveling.

• any object: PC to PC, person to person, person to object, and object to object.19

2.3.2 Human-computer interaction 2.3.2.1 Ubiquitous access to audio and video content Multimedia content must be available on any network and accessed from any suitable device to provide maximum flexibility to users. Much of the broadcast media content created and downloaded by third parties is typically read-only, non-interactive content and is often stored and manipulated on the access device. Even when content is read-only, users often want to add value to content by selectively splitting it, annotating content, and integrating multiple heterogeneous content. Locally created user content is modifiable. For content to be used in this way, it must be annotable and the media sources must be interoperable. Individual voice, video, and audio services are often unaware of each other and are sometimes not user-configurable to allow them to dynamically orchestrate more advanced services. For example, when a voice call comes in, the TV and radio are automatically paused or muted. Voice calls can be recorded on answering machines, but cannot be easily exported to other systems or converted to other formats, e.g. B. Text for a document or email to someone. Supporting such dynamic service composition requires the use of a ubiquitous network infrastructure, standard multimedia data exchange formats, dynamic service discovery, and the use of metadata. Metadata describes content, network characteristics, and associated mediation service characteristics to link and assemble heterogeneous content and enable access anywhere, anytime, on any device, in any composition.

In our model, a PC can be replaced by any other ICT system, including embedded computing systems. Objects are objects of the physical world.

Everyday applications in the virtual, human and physical world

Audio and video cluster

The computer as Hub des Clusters

red local

Processing: editing, commenting, composition Management: resources and content

amplia red

Email, Internet, Chat, VoD, VoIP

Internal communication.

user interaction

storage

Audio and video flow Figure 2.5 Audio and video cluster distributed in a home local area network with a PC as a hub

A typical group of audio and video devices is shown in Figure 2.5. Audio and video devices are often connected on a simple Webstar local area network to a PC as the center of the device network. Other home clusters may exist for home heating, lighting, and security. Most electronic devices with embedded computers do not have a network interface, e.g. B. washing machines, answering machines, vacuum cleaners, etc. Therefore, the user can only check the status or manually configure the devices through the local control interface. Smart devices and services are discussed in more detail in Section 3.1.

2.3.2.2 Widespread access to information and electronic books A personal digital calendar is a good example of a distributed information application. It can be accessed and used through a variety of devices such as PCs, set-top boxes, smartphones, and PDAs to provide partial user context to synchronize user activities in a variety of situations. The pull type interaction allows users to initiate the exchange of information, such as: B. Browsing the web in search of information and services. Push notification services are used to notify clients about events, e.g. B. news, lottery numbers to report. Surveys conducted in 2004 confirmed that the PC was still the dominant interactive device for accessing information, but the beginnings of the introduction of digital television and mobile and multi-platform access can be seen (EU, 2005). In general, PCs and various AV equipment displays and controls are not available or conveniently located in buildings and can be very cumbersome to use for daily tasks. For example, placing an ICT system in a kitchen can provide instructions on how to cook. In this case, ICT systems must be able to tolerate liquid and food spills and support hands-free interaction, as hands may be dirty or involved in other activities. As such, mobile devices are widely used to support ubiquitous access to selected content. In some cases, however, mobile devices are just as limited in terms of HCI. Electronic access to information has several advantages over paper-based systems such as newspapers, magazines, and books. Electronic access to information offers more flexible customization of text size, indexing, searching, linking to different articles, searching, commenting, editing, and enough capacity to store thousands of articles and books on a single access device. Electronic information also supports the exchange of information between applications and media, e.g. B. The text can be translated into another language, converted to speech, and sent via email.

Applications and Requirements

The PC, as a ubiquitous electronic information access device, supports both reading and writing, and can support relatively free copying of content. However, the PC suffers from a number of limitations compared to its paper counterpart. PC information devices are relatively expensive, consume a lot of power, are heavy, take time to start up, cannot be read in sunlight, cannot be controlled in the dark (enter keys are not illuminated), and are cumbersome to control with the enter keys. Paper also has the flexibility to be rolled and folded to support larger area formats, while computer screens are fixed in size and require users to scroll or zoom in and out to view larger area content. Also, many current PC software readers do not seem to be very adept at adapting the layout and column formatting of the originally published format to new formats, e.g. h to make it easier to read on display devices of different sizes. Instead, viewing often requires a high degree of cumbersome scrolling to view continuous passages of text. For this purpose, more specialized electronic book readers such as eBooks and ePapers have been developed. E-book readers20 are lightweight, slim, durable, powerful, pocket-sized devices with touchscreens that allow you to turn pages by touch. A key difference between computer screens and eBooks is the type of screen used. The screen of an e-book is designed more like paper, so e-paper21 reflects light rather than transmits it. It is readable even in direct sunlight, is equally visible from all angles, and uses a static image. There's a refresh rate of zero when the image doesn't change, so it won't drain your battery. Content can be delivered to an eBook via a wireless connection, eliminating the need to physically pick up a book or the latest newspaper. While paper can be recycled for reuse, e-paper can be reused in an even more environmentally friendly process.22 A good candidate technology for e-paper and e-books is an electrophoretic screen (Section 5.3.4 ).

2.3.2.3 Universal local control of ICT systems Computers are generally controlled through input control devices that are built into the computer and are used while the person is sitting in front of the computer. These may also have a built-in wireless remote control interface, allowing them to be controlled remotely, such as using a wireless mouse and keyboard, but still being used by a user sitting at the computer. Many devices have device interfaces, such as infrared, that allow them to be controlled by local short-range controllers23 from a few meters away. Well-known conventions are used to label buttons that map to common functions, such as changing the volume and turning the device on and off. However, ECU developers allow unusual tasks with a

Two e-book readers available in 2008 are the Reader from Sony (www.sonystyle.com) and the Kindle from Amazon (www.amazon.com). Interoperable content formats such as .epub, an XML extension defined by http://www.idpf.org/, are supported, but different digital rights systems cannot yet make them interoperable. According to Harrison (2000), the concept of the electronic book was first devised by the Frenchman Albert Robida in 1895, before electricity, television and aviation. 21 There are several technologies for creating electronic paper, some of which can use plastic, organic, and electronic substrates, making the display flexible. Electronic paper can not only be designed to read and write, but can also be used as a substrate for electronic circuitry (Berggren et al., 2001), paving the way for transferring electronic circuitry over a network to dedicated printers: replication. of electronic circuits. . 22 Reusing printed paper means recycling. Recycling involves physically transporting paper to collection points, sorting it, and then transporting it to recycled paper mills. The paper fibers are shredded before being washed and sieved to remove unwanted minor contaminants such as staples, glue, glue, ink, and plastics. The pulp is then pumped into paper rolls for reuse, pressing and drying, etc. Reuse of e-paper simply requires airborne transmission of the new content and reloading to change the molecular structure of the paper into e-paper. 23 These controllers are known as remote controllers, but are actually better described as local controllers because they can only operate locally, i. h in many cases more than 1-10 meters.

Everyday applications in the virtual, human and physical world

Combination of keys and modes with the corresponding instructions for these combinations collected in a complementary manual. These instructions can be difficult to follow as they are very short and often do not accept errors. There are a large number of local drivers, but each one is configured to control only one device, which leads to clutter and finding misplaced drivers. Two types of universal local control handheld devices that can be configured for multiple local devices have been proposed: hardware and software devices. Commercial versions of both rigid and soft universal controller devices have been released24. Both types of general purpose controllers require non-trivial manual configuration and knowledge of the manufacturer's control code for the device. Simple general-purpose hardware-based controllers have a fixed number of buttons, can control a fixed number of devices (often up to five), and often do not include a display to show status. Software-based universal device controls contain hardware buttons and a programmable button indicator, giving users more flexibility to map buttons with actions and to view status information (see Figure 2.6). Important design issues are how a driver can determine which devices are local, what features the device supports, and how to describe them. One way to do this is to allow the device to notify users about multiple web URLs by specifying the layout, model, and manufacturer URL. The universal device must then download the device information through an Internet connection and configure the device's user interface for use. This UI and device detection adds complexity and latency to device access. Software-based universal device controls can be customized by the user. They can be configured to provide a simplified user interface for casual users or a more functional one for regular users. You can customize to play only those devices that can be accessed locally. They can be customized by users. It is also suggested that the use of such universal software drivers may be

DVD

TELEVISION

Radio

play channel

canal 5

record channel

canal 4

17:38

ABC 2

ABW 3

RECORDING 4

JKL 5

OMN 6

PQR 7

THE 8

WXYZ 9

Figure 2.6 Using a soft universal local controller to interact with the washing machine, TV, DVD recorder and radio

See, for example, Philips Pronto programmable touch screen remote controls, http://www.pronto. philips.com, accessed November 2007.

Applications and Requirements

economically attractive to device manufacturers, since improving the traditional user interface of devices can be very expensive. Instead, the user interface for such complex tasks could be offloaded to a universal local driver (Roduner, 2006). The use of cell phones and PDAs as universal local controllers seems attractive due to the inherent characteristics of mobile devices, powerful microprocessors, local and remote wireless network support, and a reprogrammable screen and buttons. Studies by Roduner et al. (2007) demonstrated that the use of soft universal controllers can make simple tasks more difficult to perform than on the proprietary mobile device, but more complex tasks are easier to perform. The reason for the latter is that the mobile device can provide additional information about the state of the device, either by providing a display to non-display devices or by augmenting an already existing but smaller embedded display. Roduner (2006) considered the limitations of the mobile phone as a universal interaction device. The main limitation is that in many everyday situations, direct manipulation of the device is easier, faster, and more convenient than interacting with your handheld. Some devices may also require the user to be present at the device for security reasons, e.g. B. to control a stove or microwave. Universal controllers also present simultaneous control issues when multiple users try to control the same device with multiple controllers.

2.3.2.4 User insight and personal space personalization allows content and services to be tailored to individuals based on knowledge of their preferences and behavior (Section 5.7.4). Personalized services have several advantages, such as: B. more convenience and more relevant filtered information, but possibly with some compromise against loss of privacy. Users can customize the settings of the services, e.g. B. Each resident can access filtered content according to her preferences. Users can customize the annotation of the content and thus organize it according to their personal preferences. The same personal settings and preferences could follow the user and be shared between devices and services, freeing the user from having to manually reconfigure similar preferences on similar services. Mobile devices are often designed for personal use, for discreet use within our personal space. Mobile devices provide an obvious means for users to personalize their environment, but mobile devices do not necessarily allow sharing of the personal model with the mobile device. Section 2.3.2.3 proposed a personalization approach based on devices exporting their presentation capabilities and personalizing them on the mobile device. Nakajima and Satoh (2006) propose a model for coordinating and configuring multiple distributed home appliances using a ubiquitous personal model based on each having their own (centralized) personal home server. The personal home server can seamlessly discover and configure devices at any location. There are three general requirements for the Personal Home Server: (1) Acknowledgment of service when devices interact spontaneously; (2) ease of interoperability; and (3) maintain the privacy of personal profiles. Multiple personal spaces can also expand to shared social spaces. Some simultaneous control may be required when multiple users attempt to personally interact with the devices in conflicting or incompatible ways, e.g. For example, one user wants the music volume to be turned up, while another user wants the music volume to be turned down.

2.3.3 Human-to-Human Interaction (HHI) Applications Humans belong to multiple organizational groups, such as family, friends, colleagues, acquaintances, neighborhoods, and communities, at work, at home, and outside of work and at home. . Whereas in the past social interaction was more physical, electronically mediated HHI over a large area allows social connections to remain through voice, video and text when people are not as close together.

Everyday applications in the virtual, human and physical world

2.3.3.1 Transaction-based e-commerce and m-commerce services E-commerce enables customers to provide local and remote services and products and exchange them for money. While eCommerce might be fully automated, it is usually only partially automated. Human interaction is involved in sales negotiations, payment interactions, delivery of goods, and after-sales services. Other automated processes may be involved in product selection, shipping, and payment for products. E-commerce payments typically involve: a payment transfer from the payer to the payee or merchant in exchange for goods or services. This is an indirect money transfer from the payer's bank to the beneficiary's bank. The payment authorization is transmitted via a secure channel in the form of authorization codes, e.g. B. payment card number, payment card expiration date or personal identification number (PIN) private to the payer and given as an authorization code for the transfer of funds to the payee. It also includes an audit trail of e-commerce transaction receipts for the parties involved. M-Commerce, short for Mobile Commerce Services, are e-commerce services accessed via mobile devices. There are two types of m-commerce in terms of whether mobile users access remote services locally or whether mobile users access local services locally. Remote services and data, such as music, ringtones, and video clips, can be downloaded and purchased via the mobile phone. Payments for smaller amounts may be charged to the customer's phone bill depending on the type and size of content downloaded and a bill from the phone company to the merchant at a later date. In this way, it is not only possible to access and pay for ICT services, but also for various services in the physical world, see Section 2.3.4.2. Payment for larger transactions can be done in the same way as with e-commerce. Watson et al. proposed an extension of mobile commerce for use by UbiCom, called u-commerce. He suggested. (2002). This is the use of ubiquitous networks to support seamless, personalized communications and transactions between an organization and its various stakeholders to provide a level of value beyond traditional commerce. It is characterized by ubiquity, being available everywhere; Universality, being able to operate in heterogeneous environments (everywhere); Uniqueness, binding services with a context such as location and unison, orchestration of services that allows multiple parties to work together. There are some obvious differences between eCommerce and mobile commerce. First, e-commerce services can use a wider variety of smart devices for payment authorization, such as B. the various contactless smart cards (Section 4.4) that can contain encrypted payment authorization details. These can be swiped on or near a secure payment card reader to authorize the transfer of funds to a merchant, transmitting the same type of authorization information used for e-commerce. Second, U-commerce offers a greater focus on dynamic pricing through context-aware service delivery (Section 2.3.3.2). U-Commerce can also change social behaviors around purchasing items, as the speed and ease with which items can be purchased can exacerbate impulse buying of items that can be purchased without sufficient consideration.

2.3.3.2 Increasing the productivity of mobile people Productivity can be bottlenecked when people don't have the right information where and when they need it. Mobile users may need to be more informed as they move from A to B. Location awareness allows mobile devices to determine their position and tailor content such as local services to the position, see Section 2.3.4.1. The most common location-based devices are stand-alone map-based devices, such as SatNav (GPS satellite navigation) devices, which display location-specific positions, routes, and services through map views. . Location-based services can also be embedded in devices like mobile devices or even umbrellas that are connected to the internet and designed to make walking more fun on rainy days.

Applications and Requirements

The Pileus Smart Umbrella25 uses the bottom top of the umbrella as a screen for the screen and features an integrated camera, motion sensor, GPS and digital compass. The Pileus umbrella supports two main functions: social photo sharing and a 3D map navigation powered by Google Earth. Matsumoto et al. (2008) considered two types of display technology, using the screen itself as a screen, e.g. B. use OLED displays (Section 5.3.4.3), instead of using a lightweight video projector. The video projector was chosen to allow the use of low-cost standard screens and then add the interfaces to the screen to aid in projection, positioning, etc. If the screen cracks or breaks during the storm, only a cheap screen needs to be replaced instead of more expensive OLED screens. Furthermore, the development of micro MEMS projectors (Section 6.4.2) should theoretically lead to much lighter, smaller, and more energy efficient projectors. Business applications for mobile users include sales documents and data, database access, automation, email, and collaborative workflows. Mobile users can access mobile-based applications such as B. Agenda, calendar, and notepads. Tracking services can be configured to support automatic call forwarding and mobile-based remote workspace streaming. However, many interactions tend to be one-way. Supporting more two-way interaction in space can be beneficial, e.g. B. Field engineers and sales representatives, resulting in faster decision making based on local assessment and feedback to managers. The question arises whether access to mobile services empowers employees or enslaves them because they are always available to others, such as bosses and customers. The term “communities of practice” describes the more informal information use and task sharing that can take place among colleagues. Group practices can be set up for home activities, e.g. B. Gardening, Housekeeping and ICT Housekeeping. The acquisition, organization and harmonization of such implicit knowledge will also be a great challenge. (Shepherdson et al., 2003) developed and applied a workflow management framework that supports more decentralized and worker-centered teamwork coordination, allowing workers to plan work requests and work schedules, share work requests and work shifts, make collective decisions, work time to lengthen or shorten, and consult additional experts. More flexible, usage-based trips can also be planned and rescheduled.

2.3.3.3 Community care “Vulnerable” people at home may be supervised by friends, family and health professionals in other locations. These are then notified of events such as a subject's abnormal immobility or lack of activity. Low-cost, non-invasive technologies in the home or in residential or assisted-living facilities may allow the elderly, disabled, and others with chronic illnesses to stay in the community longer and be independent longer. to reduce their use of professional health services, as well as their demands and costs to society. There are two basic types of approaches regarding whether the subject explicitly seeks help from others, or whether the subject is monitored and modeled to allow others to anticipate when the subject will need help. An example of the above user-requested approach to care is that approximately one million people in the UK have access to hanging or pulling rope systems (Edwards et al., 2000). An example of the latter approach is the BT CAREnet study which ended in 1999.

See http://www.pileus.net/, accessed January 2008. Pileus is Latin for cap. It is now also used to describe a cloud that appears above a cumulus in meteorology. The project, an offshoot of Keio University's Okude Lab, uses pileus for both metaphors, for a physical umbrella (mushroom cap) and for cybernetic network services (cloud overlay).

Everyday applications in the virtual, human and physical world

Sensors such as passive infrared (PIR) motion sensors and magnetic proximity switches in rooms and appliance doors strategically placed around the home allow a third party, such as a family member or professional caretaker, to remotely control remote the activities and status of a person. First, a profile of normal user activity is created. The new data can then be automatically analyzed for anomalies. For example, in the CAREnet study, alarm conditions were triggered when the user did not get out of bed, was less active than usual, used kitchen appliances less than usual, and the room temperature was too low. Depending on the type of sensors used, there are many different variants of projects. However, it can be difficult to distinguish true positive and true negative events from false positive and false negative events, and to balance intrusion into a person's life to monitor a subject versus maintaining their privacy.

2.3.4 Human-Physical-World-Computer Interaction (HPI) and (CPI) Innovative ways to support mediated human-physical space interaction (Section 2.2.4) have already been presented and include Smart Tables, Smart Floors, Smart Rooms and smart tiles. Other Everyware applications are described below.

2.3.4.1 Physical environment awareness services may target specific contexts of the physical environment, such as: B. Knowledge of location, temperature, and precipitation. We can distinguish two types of location awareness: long-range mobility-based location awareness and short-range static location awareness. Proponents consider location awareness to be one of the key drivers of mobile communications, e.g. B. Tracking and navigation of people or business assets. Here the focus is on long-distance tracking of the position of a moving object relative to a target position along a pre-established path to the target. Static location awareness involves the dynamic location of services, such as a meeting place or tagged personal items, such as keys and library books, etc., within a location. Context-aware systems and applications are discussed in detail in Chapter 7. Sensors for specific phenomena in the physical world are statically embedded into specific devices and services rather than being networkable. For example, a heating system can turn on the heat when the temperature drops, light sensors can turn on the lights when it gets dark, and sprinkler systems can periodically water the garden when it doesn't rain.

2.3.4.2 Environmental (Physical) Control A cellular phone or other portable device can use a wireless connection to issue simple control commands to start or stop network-enabled devices embedded in the environment, or to access physical resources similar to controlling ICT devices locally (Section 2.3.2.3). Resources can be privately owned, e.g. B. garage door or car door, or may be provided as pay-as-you-go services. B. a beverage dispenser or shoe shine machine. In the first case, successful authorization leads directly to restricted free access. In the latter case, authorization is required to pay for the use of the service before granting access. This could involve adding a special network billing fee to dial a number associated with a product and then sending a special code to confirm intent to purchase. The network operator then forwards part of the remuneration to the service provider (Section 2.3.3.1). However, the control and reconfiguration of many of today's electronic devices is not designed for long-range remote access. These are only designed to be activated manually via a faceplate or via a proprietary short range IR remote.

Applications and Requirements

2.3.4.3 Smart Utilities Energy is required to power embedded computers and analog systems, to support various activities in the physical world, such as heating and lighting, and to support various human activities, such as traveling and eating. These power generation networks are in turn controlled by computers. Power seems to be ubiquitous in many prosperous countries, supplied by continuously operating power grids and by a mix of non-renewable and renewable energy sources. Some devices do not have internal energy storage and must remain connected to the power grid to function. Other devices may be mobile using rechargeable, rechargeable, or renewable energy sources and may operate for limited periods of time without being connected to a power grid. Mobile devices can also use self-powered systems powered by walking, arm movements, vibration, etc. The demand for energy increases. There is an incentive to increase the energy efficiency of appliances and the way they are operated to reduce waste and lower costs, e.g. It is estimated that modern power supplies (internal and external) for electronic devices are typically only 50% efficient, resulting in large power losses. Therefore, there is a lot of room for further improvements in terms of energy efficiency. Smart energy meters are networked and allow real-time monitoring of energy consumption, so customers are better informed and can regulate their own consumption. In a demand response system, customers can save money by adjusting power consumption in response to dynamic pricing signals and policies. For example, during peak periods when prices are higher, power-hungry devices might run more sparingly to save money. In direct load control systems, the electricity provider or a third party remotely controls certain devices that draw power from the customer during peak demand periods. Context-aware power devices can be turned on or off in specific ways when not in use, e.g. For example, the heating system could also detect the presence (or absence) of occupants in the building by regulating the heat. Smart grid multi-agent technology applications allow such products to act together as resources within the power system, e.g. B. Not all lights turn on when someone is nearby, just select the ones that best support that activity.

2.3.4.4 Smart buildings and home automation Automation is increasingly used in building automation, such as lighting and climate control, door and shutter control, security and surveillance systems, etc. It can also be used to control home entertainment multimedia systems, automatic plant watering, and automatic feeders. Some controllers can be surface mounted, e.g. building lights etc. They can be designed to turn on automatically when it gets dark or when motion or body heat is detected. In other cases it may be better to design systems as part of the building, e.g. B. Doors that open when people walk towards them and then close. When building a new building, control cables are usually added before the drywall is installed. These wires lead to a controller, which then controls the environment. In an upgrade situation or when installing control wiring that may be too expensive or simply impossible, there are alternatives such as powerline protocols, e.g. B. X10, Universal Powerline Bus and wireless protocols such as ZigBee or the use of standard wired PC interfaces. Home automation seems to be more widespread in the US than in Europe. Spinellis (2003) provides a good summary of the untapped potential of today's home control systems, such as heating, lighting, garden watering, media players and recorders, which are often used alone.

Everyday applications in the virtual, human and physical world

Device mode and controlled by independent local closed system controllers, each with its own wireless or powerline interface. There are a large number of IR remote controls that have inconsistent interfaces for performing the same control operations. Thus, for the devices described, it is difficult to determine what actions are possible at any time, the conceptual model of the system and the current state are hidden from the user, and there are no natural associations between a user's intentions, the required actions. and the effect of the result. Spinellis (2003) proposed using a PC hub to add existing home control, "infotainment" (playback of stored music files), security (serial connection to security alarm devices), and communication technologies (telephone interfaces for connect to a door entry system). Arens et al. (2005) report how wireless sensor network technology could influence future building design and operations. Flexible sensor placement and increased density of multiple sensor types can significantly improve the energy efficiency of buildings. The Future_Home project enabled people in the home to move, control, communicate and enjoy AV entertainment without being aware of the underlying technologies or networks (Alves et al., 2004). A key motivation is that today's home is not well equipped to keep up with rapid technological changes and recent sustainability concerns. It is an inflexible space in a dynamic world, unable to respond to the ever-changing demands of its occupants. This is partly due to the characteristics of the metasector of a habitat, where traditionalism, fragmentation, lack of innovative capacity and lack of communication characterize many of the actors involved. Alves et al. pointed out the need to establish a methodology for the construction and maintenance of an experimental house of the future. They also noted that the technology required to improve the habitat metasector requires action on many fronts, such as construction methods, production technologies, engineering solutions for structural components and systems, and investment in collaborative activities with research institutions, universities, and companies. . They state: The nature of project-based work in the construction industry means that companies must manage highly complex networks of innovation interfaces. Therefore, construction can be viewed as a complex systems industry in which there are many hierarchically organized, interrelated and custom elements, with small changes in one element of the system leading to large changes elsewhere.

2.3.4.5 Smart home environment and smart furniture Various smart environment devices can adapt to human activities. Doors, lighting, faucets, and ventilation can be designed to sense, activate, and adapt to human presence. An oft-cited example of a smart environment is a refrigerator that knows what ingredients are stored in it. You may also know about external system applications, such as journals, that are needed for meals that are scheduled later in the week. The smart fridge behaves like an inventory control system, automatically detecting low quantities or expired foods such as fresh vegetables and milk. This is easy to use when there is only one pending order; however, the containers can be large. There is the complication of how the remaining amount is recorded, e.g. B. Use a transducer to weigh incoming food, but you also need to know the weight of the container. The system is empowered to act on the context to order new groceries when authorized. Policies may include not activating the Order New Groceries promotion when consumers are away, or not being able to order more groceries than the store size, even if there is a volume discount. Preferences for organic or skim (low fat) milk can be taken into account when ordering. The order does not lead to a fully automatic delivery of groceries to the warehouse. This requires human assistance and human coordination for delivery. Ordered food could be marked in such a way that the customer can keep track of their orders. So if you knew the delivery schedule, you could estimate when it would be delivered to better plan to be physically on site when the shipments arrive.

Applications and Requirements

A ubiquitous home environment could be designed to support healthy eating and weight management for its residents. This could be supported through the use of the following networked components: a central distributed ICT system that supports screens and mobile devices of various sizes and with flexible positioning; refrigerator(s); multiple storage areas that recognize what food and drinks they contain; Food and human scales and portable scanners. Scanners are used to identify food and beverages on packaging, this can include a barcode, but not all foods can have barcodes. The scanners can be used to identify and select healthy foods to buy at home and in grocery stores such as supermarkets. The system can also search for text labels, expiration dates, food ingredients, and ingredient weight percentages. The system could monitor the daily and weekly amount of food consumed, the number of calories consumed, and the weight in grams of fat, salt, and sugar on a daily basis. You could proactively suggest and support eating habits for a balanced diet, e.g. B. trying to eat five pieces of fruit or vegetables a day. You could keep a list of unhealthy or undesirable food ingredients that cannot be selected: hydrogenated oils or trans fats, monosodium glutamate, etc., and foods that are not allowed for religious reasons. You could support food inventory control to minimize food waste and get new supplies when food expires. Smart objects in the home environment, such as mugs, can be designed to sense their physical state and autonomously map sensor readings onto a virtual computer model. Simple electronic tags, such as RFID tags, could easily be attached to objects. More complex integrated network control devices and sensors can also be added, e.g. the MediaCup (Beigl et al., 2001). MediaCup was not developed for any specific application; however, various applications have since emerged, for example door sensors can be used to determine the accumulation of hot cups in a room to make inferences and indicate gatherings. Users can be tracked through their mugs. Users can be warned if they pick up a cup that is too hot to drink. Smart chairs such as the SenseChair (Forlizzi et al., 2005) can be designed to collect and adapt to information about the behavior of a seated person. The need for direct user input into this chair is designed to be limited to turning the chair on and off. The chair can determine the distribution of body weight across the surface of the chair as measured by pressure sensors and correlate this with the time since the last significant body movement and the time of day to indicate, for example, that the participant is sitting down. slept in the chair after dark, prompting the chair to respond accordingly. Smart watches can be used not only to display the current time, but also to provide contextual information, e.g. B. where the people are at the time or what the weather is like at the time. For example, the Whereabouts Clock26 (Sellen et al., 2006) is essentially a situated screen. It is a persistent display of information at a glance that can be shared publicly or privately. This application is location aware by determining the location of a person's mobile phone, each person's location from a fixed set of static locations is indicated by the position of a clock hand. Smart mirrors can automatically optimize the field of vision, e.g. B. The mirrors on the passenger side of the car can automatically point down to see the curb when you back up to the curb. Mirrors can be intelligent because they can be connected to actuators and sensors in the physical world, e.g. For example, mirrors containing cameras and sensors can be attached to toothbrushes to automatically guide or display the view to help users identify which parts of their teeth need extra cleaning. Glass and mirrors can be used as extended screens, e.g. For example, a vehicle's glass windshield can display the speed of the vehicle so the driver doesn't have to take their eyes off the road.

26 The idea for a watch that would show people's location instead of the time was pioneered by J.K. Rowling's Harry Potter Stories. The Weasley family has a magical clock, first introduced in Book 2, Harry Potter and the Chamber of Secrets, with hands for each family member showing the location or condition of it.

Everyday applications in the virtual, human and physical world

Sixty-five

Park et al. (2003) discuss a number of smart home ideas as follows. The smart wardrobe requests the weather forecast digitally so that the user can conveniently and properly coordinate their clothes with the outside environment before leaving the house. A smart bed can be programmed to remember your preferred sound, smell, light and temperature settings to gently awaken all your senses and start each morning off right. A smart pillow can read you as many books as you choose before bed and play your favorite music to lull you to sleep when you start to feel sleepy. As soon as your body enters a deep sleep, it automatically checks your sleep status and quality, gradually reducing the music volume accordingly and finally turning it off completely. A smart mat placed at the entrance of each home can be used to detect users' body weight and footprint, which can enable the smart mat to distinguish and recognize who is stepping on the mat. A smart sofa can enhance your experience of watching TV or playing video games. Depending on the images and sounds on the screen, vibrations are used to enhance the visual experience in action scenes.

2.3.4.6 Intelligent urban furniture Urban furniture refers to the installations installed on roads and paths, such as benches, bollards, lights, traffic lights, road signs, public transport stops, garbage cans, taxi ranks, public toilets and fountains. . For example, in some cities, bollards can be remotely raised and lowered from buses, providing secure access control to lanes reserved for buses but not other vehicles. Street furniture often uses solar panels to supplement the power supply. They are networked and can house other devices such as video cameras and sensors. Traffic signals can also adapt to traffic patterns instead of being timed.

2.3.4.7 Intelligent vehicles, transport and travel Embedded computing systems are increasingly being used in vehicles. This helps to improve its operations, such as B. Automatic control or provision of assisted control when the driver still has some control over antilock brakes, airbag inflation, cruising speed, vehicle climate, prevention of collisions through automatic braking, etc. Computer positioning systems also allow remote tracking of vehicles and merchandise and inform stakeholders about their schedule. Some transportation systems can be automatically guided and controlled along the tracks without a driver.27 In some cases, the stations themselves are also unmanned. Train, plane and bus connections can be arranged in such a way that waiting passengers are informed about the status of arriving and departing vehicles. Current traffic information can now be distributed much better and retrieved much more conveniently, e.g. B. through a mobile device and at intermediate points of traffic, such as bus stops. Travel authorization tickets are also smarter. In some cases, e-tickets can be requested, paid for, and issued remotely. Smart tickets can also be designed to be prepaid and support contactless verification at passenger boarding and disembarkation points.28 Passengers and drivers of smart mobile vehicles, such as trains, ships, and planes, are increasingly accessing the Internet. Once connected to the Internet, not only Internet data can be accessed

For example, the Docklands Light Railway in East London, which opened in 1987. Smart tickets often contain RFID chips that can be read from a short distance by swiping the card, e.g. B. The Phillips MiFare ticket is used in London. where it is called Oyster Card and in other cities around the world. This system uses the industry standard ISO 14443A for contactless radio frequency smart cards. The card is also swiped at exit to set the fare in proportion to travel time. 28

Applications and Requirements

the vehicle, but vehicles can also be monitored and various environmental information generated inside the vehicle via embedded computers and sensors can be uploaded to the Internet29. Since 1995, the InternetCAR (Internet Connected Car Research) project at Keio University, Japan, has been investigating how vehicles can transparently connect to the Internet (Ernst et al., 2003). Three separate networks of vehicle data sources are identified: (1) information services such as audio streaming and navigation services; (2) passenger-controlled devices (some lights, power windows, etc.); (3) Devices that operate the vehicle (engine and brakes, sensors that monitor the air pressure in the tires or the amount of fuel left in the tank, etc.). The focus of the InternetCar project was the mobility of cars modeled as mobile computing nodes within a wireless network, and considering the ability of some cars to act not only as providers or users of services, but also as part of them as mobile routers. to function as a large-scale ad hoc network.30

2.3.4.8 Widespread Play and Social Physical Spaces When making friends in the local area, a "friends meetup" invites the various parties to meet at the merchant's local premises, such as a store. B. a coffee to meet. Vendors like the coffee shop may even sponsor the service to make it free for users, presumably because there is a higher chance that this will increase trade. The types of business locations suggested as meeting points could depend on the dynamic personal preferences of the parties involved. Local dealers who require customers to be present in person can advertise their offers on the site. This requires services to strike a careful balance between providing timely, location-based, and helpful notification services, and delivering floods of unwanted notifications from many competing providers that result in users being overwhelmed with notifications. . Notification services can be message-based or context-based when a user searches for a specific service. Privacy concerns aside, there are plenty of interesting social and economic topics here, e.g. B. Dynamic market equilibria could be achieved when resident customers in a store receive a better offer from a nearby competitor and threaten to leave unless the current or resident business makes them a similar or better offer. Games are an important type of social and interactive entertainment and application. Traditional or pre-electronic games consist of two types of interactions, human-world-physical interaction (HPI) and human-human interaction (HHI). HPI refers to the movement of people and the interaction between them, e.g. B. Sports or people moving various physical objects, tokens, on a board, taking cards, etc. HHI involves competing with others to win a game. On the contrary, electronic games create the illusion of being immersed in an imaginative virtual world. Computer games can be more interactive than traditional games and can be designed with an optimal level of informational complexity to provoke and engage players. Electronic games often focus more on HCI than CPI and HHI. Electronic games tend to virtualize people so that they immerse themselves in the game and interact in a virtual ICT environment. There seems to be a stark social divide between young gamers and older non-gamers. An analysis of the requirements of existing game consoles indicates a significant lack of appeal of video games to a broader audience. The game's control interface (D-pad interface, q dual joystick, and four-way toggle switch) is often considered difficult to use by newcomers or non-players. Game console innovation often focuses on developing more powerful and visually stimulating virtual games.

Currently, a typical car contains around 120 sensors and 50 different embedded computer systems. In theory, all road vehicles could be configured to form the world's largest ad hoc computing network, with millions of computing nodes interconnected in one region. 30

discussion

Environments (Grossman, 2006). This has led one game console manufacturer, Nintendo, to scrap the typical game console interface and replace it with a simpler interface called the Wii, which resembles a magic wand. This is part laser pointer and part motion sensor so you can monitor where you're pointing and how fast you're moving. It is also designed to recognize gestures. Pervasive games create various social activities and games that attempt to harness the potential of combining physical space, the use of physical objects, and a sense of physical space augmented by a virtual space (Magerkurth et al., 2005). Kampmann Walther (2005) gives a good overview of the variety of different types of games that are widespread. These include: 31

• Mobile games: taking into account changes in the relative or absolute positions of players and objects in the game rules;

• Location-Based Games – Include relative or absolute locations in the game rules, e.g. B. Treasure Hunt;

• Augmented Reality Games and Mixed Reality Games: These are interesting approaches to the

•

Creating game spaces that try to integrate virtual and physical elements into a perceptual game world that can be experienced in an understandable way, e.g. B. Electronic magic wands with gyroscopes can detect orientation and movement and merge the movements of these players into a virtual landscape; Adaptronic Games: Applications and information systems simulate life processes observed in nature. These games are integrated, flexible, and generally consist of "touch parts" that oscillate between virtual and real space.

Ubiquitous games support UbiCom distribution properties. Players are also often mobile. Players' actions take place in the physical world and they can be aware of the location of themselves and other players. Additionally, games are designed to persist between player sessions and can support transmedia. Transmediality refers to the use of concepts that transcend individual media. The interaction of multiple media, distributed across vast networks and accessible through a variety of devices, is a good example of media communicating in circular rather than linear ways. These media convey information, entertainment, games, dramatizations, and character sketches in a continuous cycle of coupled quotes and usage codes that can be timely adjusted and functionally modified (Kampmann Walther, 2005).

2.4 Discussion 2.4.1 Achievements of the first projects and current status The first projects mainly focused on UbiCom's three basic requirements of supporting distributed systems, especially for mobile users, a variety of iHCI and context awareness. These particular early projects tended not to focus on the UbiCom systems, which were designed as intelligent, autonomous systems.

2.4.1.1 Smart Devices The focus of the early projects at PARC and Olivetti, which began in the late 1980s, was more on the basic design of smart device models for tabs and pads. could you support

Pure VR gaming was excluded as a ubiquitous gaming subspecies because UbiCom's focus is on media rather than VR. Ubiquitous games are synonymous with ubiquitous games.

Applications and Requirements

Communication and location awareness for mobile users. In the late 1980s there were no commercial mobile ICT devices or widespread wireless networks. Apple Inc. manufactured and marketed the first handheld computer or personal digital assistant (PDA) in 1993 called the Newton MessagePad, often referred to as the Apple Newton. Although 200,000 Newtons were produced, the system had several cost and design issues and Apple discontinued the product in 1998. By the late 2000s, mobile devices such as phones (tabs) and laptops (pads) became widespread and networks Wireless networks were widely available that supported routing of cellular data communications to users wherever they were. Today's tab-sized mobile devices use touch screens and support gesture recognition, for example, to allow users to scroll through a photo album and zoom in and out of images. An even more common application of smart tabs than their use for mobile personal communication devices is their use as smart cards32 to support a variety of applications such as driver's licenses, insurance information, chip and PIN credit or debit bank cards. , and travel and admission tickets. (clause 4.4) . In the early 1990s, PARC developed electronic whiteboards that allowed users to edit text and graphics together. They have now become commercial products. Their use as part of the Classroom 2000 project in 1995-1998 was first reported by Above (1999) and they are now routinely used in many educational institutions to support distance learning. There are also some interesting prototypes of tab-sized mobile smart devices that allow them to act as dashboards, e.g. B. Microprojectors (Section 6.4.). Electronic boards are designed to be used as large vertical displays to disseminate information to larger groups led by a single facilitator. Much of the group is far away and doesn't have an easily accessible method of interacting with the screen. The DataTiles electronic table is similar to the electronic whiteboard in that it is designed for collaboration. However, electronic tables are horizontal and designed to be used by smaller groups seated around the table.33 Any member of the group can interact with the table. There doesn't necessarily have to be a leader or mediator, so group members may need some form of field control to prevent multiple conflicting actions from being taken. This particular type of electronic table also uses physical objects as offerings to interact with the table. Weiser (Section 1.4.1.1) originally characterized devices by their form factor in terms of tabs (centimeter-inch size), pads (decimeter-foot size), and circuit board (meter/yard size). This characterization of tab, pad, and board-sized UbiCom devices should be expanded to include the use of micro- and nano-sized (powder-sized) devices and non-planar surfaces and volumes. Today, robots have a variety of uses. These applications are mainly industrial, but there are also more and more social and entertainment applications. Robots can be used to help people in the home environment by acting as intrusion detectors or digital pets.34

2.4.1.2 Smart environments of the physical world Although physical environments are getting smarter, this is happening gradually. Smart home environments are gradually acquiring more wirelessly connected devices such as speakers

Whereas around 70 million smartphones were in use in 2006, around 10 billion smart cards were in use. In 2007, Microsoft announced a Surface Computer initiative. Surface computers can transform an ordinary table into a dynamic surface that supports interaction with various forms of digital content through natural gestures, touch, and physical objects that recognize physical objects, from a paintbrush to a cell phone, enabling direct and practical control of the content. such as photos, music and maps. It is suggested that these be placed in hotels and homes where social interactions take place on a routine basis. 34 The IFR International Federation of Robotics Statistics Division predicts that 3.6 million home and leisure robots will be sold between 2006 and 2010, see http://www.worldrobotics.org/index.php, accessed July 2010. 2007. 33

discussion

Systems, bells, motion detectors for security systems, etc. Medium term (next five years), if not longer term (next 10 years). Supporting seamless interoperability in the physical environment faces many technical challenges. The physical environments still tend to be pretty passive and goofy in general. In addition to the usefulness of the applications themselves, cost, reliability, low maintenance, and security will be important factors in integrating devices into physical environments. There are two main types of intelligent physical environments: embedded computing systems and amorphous computing systems. Advances in microelectromechanical systems (MEMS) are a key enabler for smart environments, leading to the availability of much smaller, low-cost, low-resource, and low-maintenance amorphous computing devices (Section 6.4.4).

2.4.1.3 Context awareness and service awareness Location determinism tends to be supported primarily as stand-alone devices and services that are not easily interoperable. Location tracking is also built into some mobile devices, such as phones, cameras, and cars. However, these are primarily intended for outdoor use. Systems for indoor use are available today, e.g. B. based on trilateration using WLAN transceivers, but there is no single standard in routine use to support indoor location detection. Many services support some level of user knowledge. Users can configure devices and services to support persistent personal preferences. Users can choose to have the services store their personal information, such as their home or work address and payment information, so that they do not have to repeat each session for the same service. The Services may also be interoperable to share personal preferences and information. Service discovery of local network resources is weak, and discovery of other local environment resources is still virtually non-existent, so much of Cooltown's insight is not routinely available. This is partly because smart environments are not yet mature or widespread with a broader range of environments available. There are a variety of reasons for this, such as B. the variety of support required, cost, and the ability to secure fortifications.

2.4.1.4 Implants and wearable smart devices Wearable smart devices are still in their infancy. Devices can be used as watches, headphones, glasses, belts, and clothing. An early focus of Mann's work was to enable a cyborg logger or glogger35 to continuously record, process, computationally interpret, and share everyday personal life.36 Earlier versions, in the form of larger head-mounted audiovisual devices, were more intrusive, but later versions in the form of data glasses make them less intrusive. Several prototypes of new types of smart glasses are available. Glasses can consist of an extremely thin layer of liquid crystal sandwiched between two panes of glass, giving smart glasses the ability to switch between near and far vision and assisting in autofocus, allowing people who currently They wear bifocal and multifocal lenses. Smart glasses can also be designed as dual LCD screens to display videos instead of looking through them.

Mann's cyborg community has grown to over 30,000 members, see http://glogger.mobi, accessed February 2008. 36 As those who watch the current trend in reality TV shows know, a real challenge is the sheer volume from (manual) An edition It was necessary to filter out everything mundane and leave the remarkable facts.

Applications and Requirements

A second important use of smart glasses is to act as a heads-up display, or HUD, allowing pilots and drivers to focus their attention on what's going on around them and avoid looking at their instruments too often for important information. . The helmet-shaped HUDS contain a specially coated lens that reflects the monochromatic light representing digital data and annotations, while allowing all other wavelengths of light to pass through, creating an overlay image. These were first demonstrated by Sutherland in 1968 and are currently used on some military aircraft. Fixed HUDs, which require the user to look through a display element attached to the vehicle's fuselage or chassis, are also available on some current road vehicles, e.g. B. They can show the speed on a part of the windshield. Laptops are based on smart headphones, headphones, gloves, etc. Headphones are also available, which act as a directional amplifier and are useful for assisting the hearing impaired. Several watches37 have been marketed to act as personal communications and information managers, but they appear to be heavy-duty in construction and have yet to capture the attention of users. Unlike a cell phone, which can be operated hands-free, a smartwatch is worn on one hand and can only be operated with one hand by the other hand. You may need to bring it closer to your eye to see detailed information. As such, they are not compatible with all of Mann's proposed portable computing requirements; do not meet the criteria for constancy of interaction (see Section 2.2.4.5). A wireless, e.g. B. Bluetooth headsets paired with a mobile phone can be seen as a type of laptop: this system meets Mann's three criteria (Section 2.2.4.5) that define laptops. Smart headphones instead of smart glasses might be more successful precisely because they augment reality rather than convey it. Because sight is the dominant sense in that we use it to move around and gather information about the physical world, the electronic mediation of sight can have significant side effects on our natural use of that function. People can find it difficult and tire quickly when they have to constantly shift their attentional focus from a virtual visual view to a normal physical view of the world. Different colored contact lenses can improve human vision even for people with normal vision. An amber tint can improve visibility of moving targets like baseballs and tennis balls. A grayish-green tint can enhance the dips and curves on a distant golfer's green. One important application of wearable computing is virtual reality electronic gaming, in which players can wear gloves, headsets, headsets, and body suits to support multi-modal interaction in the environment. Implants are routinely placed in some animals to aid in authentication and traceability. Implants are routinely used in humans for medical reasons such as B.: pacemakers to support and regulate the electrical activity of the heart, hearing aids; Micropumps for pain relief, neurological dysfunction and even weight control. The key features of implantable medical electronics are their unique combination of extremely low power consumption, high reliability (Gerrish et al., 2005) and biocompatibility requirements. This requires significant iterative design to result in implants that meet these requirements. It is therefore not surprising that the prototype nerve implant studied by Warwick ultimately failed. A second use of implants in humans is to give us abilities beyond those of other humans. As Warwick (2003) points out, this raises ethical questions about how far research should go and whether it is good or bad to 'evolve' humans in a technical rather than a biological way (the post-human model). , Section 5.4.1). Balancing the advantages of using such implants in humans for technical reasons, such as implicit authentication and authorization, against the disadvantages, such as the increased long-term risk of malfunction and rejection by the body, and possible follow-up and loss of privacy for the individual, should be considered and carefully evaluated.

For example, "Swatch the Beep" in the mid-1990s, Timex DataLink powered by Microsoft, Fossil Palm WristPDA, Casio Databank, and the Microsoft SPOT wristwatch in 2003.

references

EXERCISES 1. In terms of the six shapes of UbiCom: Tabs, Pads, Boards, Dust, Skin, and Clay, how would you classify the vibrating string shape of the Calm Computing artifact? 2. Define Mann's three criteria for laptops. Analyze whether or not devices such as laptops, cell phones, and wristwatch information devices meet these criteria. 3. Discuss how a UbiCom system can be designed to support Abowd and Mynatt's (2000) design principles to support everyday informal activities. 4. What is ambient intelligence? Give a couple of examples. Does Smart Environment Interaction really need intelligence? 5. Think about how the Cooltown model could be used to gain advantages in the workplace or at home. 6. Compare and contrast the design principles of Abowd and Mynatt (2000) with the design principles of Johanson et al. (2002) for interactive workspaces. 7. Compare and contrast Weiser's idea of Calm Computing with Philips Homelab's ideas to support the need to belong and share experiences, the need for excitement, excitement, and relaxation, and the need to balance and organize our lives. 8. Imagine your own specific UbiCom devices and discuss their benefits and design challenges. 9. Define your own triple play, quad play, pentad play, etc. service packages. together with suitable UbiCom access devices and justify their use. (More exercises are available on the book's website.)

References Aarts, E. (2003) 365 Days of Ambient Intelligence Research at HomeLab. Retrieved from http://www.research. philips.com/technologies/syst_softw/ami/vision.html; accessed December 2007. Above, GD (1999) Classroom 2000: an experiment with the implementation of a living educational environment. IBM Systems Magazine, 38(4): 508-530. Addlesee, M.D., Jones, A., Livesey, F. & Samaria, F. (1997) The ENT Active Floor. IEEE Personal Communications, 4(5): 35-44. Alves, J., Saur, I. and Marques, M.J. (2004) Imagining the house of the future: a multisectoral and interdisciplinary approach to innovation. Paper presented at the E-Core Conference, Maastricht, The Netherlands. Arens, E., Federspiel, C.C., Wang, D., & Huizenga, C. (2005) Ambient Intelligence Research at HomeLab: User Experience Engineering. In W. Weber, J.M. Rabaey and E. Aarts (eds.) Ambient Intelligence. Berlin: Springer Verlag, pp. 63–80. Beigl, M., Gellersen, H. W. & Schmidt, A. (2001) Mediacups: Computer-assisted experiences with the design and use of everyday objects. Computer Networks, 35(4): 401-409. Berggren, M., Kugler, T., Remonen, T., Nilsson, D., Miaoxiang Chen, and Norberg, P. (2001) Paper electronics and electronic paper. In Proceedings of the First IEEE International Conference on Polymers and Adhesives in Microelectronics and Photonics, pp. 300-303. Brumitt B, Meyers B, Krumm J, Kern A and Shafer S (2000) EasyLiving: technologies for smart environments. In Proceedings of the 2nd International Symposium on Portable and Ubiquitous Computing, HUC-2000, Bristol. In Lecture Notes in Computer Science, 1927: 97–119. Bushnell, N. (1996) Relationships between entertainment and computer business. ACM Notices, 39(8): 31-37. Edwards, N., Barnes, N., Garner, P. et al. (2000) Lifestyle monitoring for assisted independence. BT Technology Magazine, 18(1): 64-65. Ernst, H.A. (1961) MH-1 – A computer controlled mechanical hand. PhD thesis, Massachusetts Institute of Technology. Available at http://hdl.handle.net/1721.1/15735; Consulted in December 2007.

Applications and Requirements

Ernst, T., Uehara, K. and Mitsuya, K. (2003) Network Mobility from the InternetCAR Perspective. In Proceedings of the 17th International Conference on Advanced Information Networks and Applications (AINA'03), pp. 19-26. EU (2005) Benchmarking Report on the Information Society. Available at http://europa.eu.int/information_society/activities/atwork/hot_news/publications/index_en.htm Accessed September 1, 2006. J. Forlizzi, C. DiSalvo, J. Zimmerman, Mutlu, B. and Hurst, A (2005) The SenseChair: the recliner as a smart aid for the elderly. Proceedings of the Design for User Experience Conference, DUX05, No. 31. Frost, GP (2003) Assessing Smart Dust. Computing in Science and Technology, 5(6):6–9. Garlan, D., Siewiorek, D.P., Smailagic, A., et al. (2002) Project Aura: Towards distraction-free pervasive computing. IEEE Pervasive Computing, 1(2): 22-31. Gerrish, P., Herrmann, E., Tyler, L., & Walsh, K. (2005) Challenges and limitations in the design of implantable medical integrated circuits. IEEE Transactions on Device and Materials Reliability, 5(3): 435-444. Gershenfeld, N. (1999) When things start to think. London: Henry Holt & Co. Greenfield, A. (2006) Everyware: The Dawning Age of Ubiquitous Computing. Harlow: Pearson Education. Grossman, L. (2006) A game for all ages. Time Magazine, 167(20):32-35. Harrison, BL (2000) Electronic books and the future of reading. Computer Graphics and Applications IEEE, 20(3):32-39. Hill J, Szewczyk R, Woo A et al. (2000) System architecture directions for networked sensors. In Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems, Boston, MA, pp. 93–104. IFR (2007) 2006 Global market for robots. Statistics department of the IFR (International Federation of Robots). Retrieved from http://www.worldrobotics.org/index.php, November 2007. Ishii, H. and Ullmer, B. (1997) Tangible Bits: Towards Seamless Interfaces Between Humans, Bits and Atoms. In Proceedings of the Conference on Human Factors in Computing Systems (CHI '97), Atlanta, USA, pp. 234-241. ISTAG (IST Advisory Group) (2003) Advisory Group for the European Community Technology Program for the Information Society. Ambient Intelligence: From vision to reality. Retrieved from http://www.cordis.lu/ist/istag.htm. Article from May 2005. Johanson B, Fox A and Winograd T (2002) The Interactive Workspaces Project: experiences with ubiquitous computer rooms. Pervasive Computing, 1(2): 67-74. Kampmann Walther, B. (2005) Atomic Actions: Molecular Experience: Generalized Game Theory. ACM Computers in Entertainment, 3(3): Article 4B. Kidd, C.D., Orr, R., and Abowd, G.D. (1999) The conscious home: a living laboratory for ubiquitous computing research. In Proceedings of the 2nd International Workshop on Cooperative Buildings, Information Integration, Organization and Architecture. Lecture Notes on Computer Science, 1670: 191–198. Kindberg T, Barton J, Morgan J, et al. (2000) People, places, things: web presence for the real world. In Proceedings of the 3rd IEEE Workshop on Mobile Computing Systems and Applications, pp. 19-28. Kintsch, A. & dePaula, R. (2002) A framework for assistive technology adoption. In SWAAAC 2002: Enabling Learning Through Assistive Technology, pp. 1–10. Krumm, J., Cermak, G., & Horvitz, E. (2003) RightSPOT: A new sense of location for a smart personal item. Lecture Notes on Computer Science, 2864: 36–43. Magerkurth C, Cheok AD, Mandryk RL & Nilsen, T. (2005) Ubiquitous gaming: bringing computer entertainment back into the real world. ACM Computers in Entertainment, 3(3): Section 4A. Mann, S. (1997) A historical account of the "WearComp" and "WearCam" inventions developed for personal imaging applications. In 1st International Symposium on Wearable Computers, pp. 66-73. Matsumoto, T., Hashimoto, S., & Okude, N. (2008) The Embedded Web: Embedded Web Services Interaction with an Umbrella for Augmented Urban Experiences. Computer Animation and Virtual Worlds, 19(1): 49–66. Maxwell, C. (2000) The future of work: understanding the role of technology. BT Technology Magazine, 18(1): 55-56. Murakami, T. (2004) Pervasive Connectivity: Business Opportunities and Strategic Issues. Available from the Nomura Research Institute home page, http://www.nri.co.jp/english/. Accessed September 15, 2006. Nakajima, T. and Satoh, I. (2006) A software infrastructure to support spontaneous and personalized interactions in home computing environments. Personal and Ubiquitous Computing, 10(6): 379-391. Park, S.H., Won, S.H., Lee, J.B. and Kim, S. W. (2003) Smart Home: digitally designed life. Personal and Ubiquitous Computing, 7 (3-4): 189-196. Rekimoto, J., Ullmer, B. & Oba, H. (2001) DataTiles: A Modular Platform for Mixed Graphics and Physical Interactions. In Conference on Human Factors in Computer Systems, CHI2001, Seattle, USA, pp. 269-276.

references

Roduner, C. (2006) The mobile phone as a universal interaction device - are there limits? In Proceedings of MobileHCI '06 Workshop on Mobile Interaction with the Real World, Espoo, Finland, pp. 30-33. Roduner, C., Langheinrich, M. & Floerkemeier, C. (2007) Mobile phone device operation: strengths and limitations of a universal interaction device. Lecture Notes in Computer Science, 4480:198–215. Rogers Y, Price S, Fitzpatrick G et al. (2004) Ambient Wood: Designing new forms of digital augmentation for outdoor learning. Proceedings of the Conference on Interaction Design and Children: Building Community, pp. 3-10. Ruyter B de, Aarts E, Markopoulos P and IJselstein W (2005) Ambient intelligence research at HomeLab. In W. Weber, J.M. Rabaey and E. Aarts (eds.) Ambient Intelligence. Berlin: Springer Verlag, pp. 50–61. Sellen A, Eardley R, Izadi S and Harper R (2006) The Whereabouts Clock: Early Testing of a Location Awareness Device. At the Human Factors and Computational Systems Conference, CHI '06. Montreal, Canada, pp. 1307–1312. Shepherdson, J. W., Lee, H. & Mihailescu, P. (2003) mPower: A component-based development framework for multi-agent business process support systems. BT Technology Magazine, 21(4): 92-103. Spinellis, D. D. (2003) The oven of information: consolidated home control. Personal and Pervasive Computing, 7(1):53–69. Stanford, V., Garofolo, J. Galibert, O., Michel, M., & Laprun, C. (2003) The NIST Meeting Room and Smart Space Projects: Signals, Acquisition, Annotation, and Metrics. Want, R., Hopper, A., Falca˜o, V. & Gibbons, J. (1992) The Active Badge Location system, ACM Transactions on Information Systems, 10(1): 91-102. Ward, A., Jones, A. & Hopper, A. (1997) A new location technique for the active office. IEEE Personal Communications, 4(5): 35-41. Warneke, B., Last, M., Liebowitz, B., and Pister, K.S.J. (2001) Smart Dust: Communicating with a cubic millimeter. Computer Science, 34(1):44-51. Warwick, K. (2003) A Study of Cyborgs. Ingenia, Magazine of the Royal Academy of Engineering, 16:15-22. Watson RT, Pitt LF, Berthon P and Zinkhan GM (2002) E-commerce expands the marketing universe. Journal of the Academy of Marketing Sciences, 30(4): 329–343. Weber, W. (2003) Ambient Intelligence - Industrial research on a visionary concept. Proceedings of the 2003 International Symposium on Low Power Electronics and Design, ISLPED '03, pp. 247-256. Weiser, M. and Brown, J. S. (1997) The next era of silent technology. In P Denning and R Metcalfe (eds) Beyond calculus: the next fifty years of computing. Berlin: Springer-Verlag, pp. 75-93. Weiser, M., Gold, R., and Brown, J.S. (1999) The origins of ubiquitous computing research at PARC in the late 1980s. IBM Systems Magazine, 38(4): 693-696.

3 Intelligent Devices and Services 3.1 Introduction Intelligent user devices are driven by increased performance and cost-effectiveness to integrate powerful embedded resources with significant data processing and storage and access to network bandwidth across a variety of devices static and mobile in different forms, such as dust, tokens, cards, blocks and boards (Section 1.4.1). Smart devices in this chapter focus on chip, card, and pad-sized devices where the locus of control for user interaction is within the device.

3.1.1 Chapter Summary This chapter is structured as follows. First, the main features of smart devices as a means of providing an embodiment of smart services and the views, abstraction and virtualization of them are considered. Various architectural models for UbiCom systems are discussed (Section 3.2). The service access and delivery lifecycle is described below (Section 3.3). Finally, operating system support for running services is discussed (Section 3.4).

3.1.2 Characteristics of smart devices and services Smart devices are characterized by the ability to run multiple applications, possibly simultaneously, support varying degrees of mobility and customization, support intermittent remote access to services, and operate within the constraints of local resources (see Table 3.1). . Smart devices are often owned, operated, configured and controlled by individual human users, e.g. etc. Some of the more complex devices, such as laptops, can be seen as aggregates of multiple smart devices.1 Like the ability to do so

A laptop is a complex device consisting of several embedded computers, e.g. contains a microprocessor to control data transfer to and from on-chip memory types ROM and RAM), etc.

Stefan Poslad

Smart devices and services

Table 3.1 Characteristics of access to the service used by smart devices Function

requirements

Design

Common features for smart devices, environments and interaction Multiple applications accessed through a variety of devices Easy service access Minimal setup and maintenance

Shared resources and services

Devices run multiple local processes; act as portals to access pre-configured sets of multiple remote services Services must be easy to start and operate Access device to local resources and remote services.

Supports access to resources by multiple users.

Supports different combinations of local and remote processing (Section 3.2.1) Modularization and abstraction of ICT resources (Section 3.2.1) Auto-discovery networks, services, etc. (Section 3.3) Use of devices, self-management of public services, etc. models (section 12.3.2) concurrency control (section 12.3.3)

Internal characteristics of the smart device Open, interoperable, heterogeneous

Users prefer a variety of service instances, access devices, and networks

Dynamic service delivery

Discover, select, compose and access services anywhere, anytime, anywhere Relevant range of mobile services and users depending on the application

mobility

Ephemeral remote service access

Intermittent access is due to network errors, dynamic access to the service, concurrency, updates, etc.

Local resource constraints

Some types of embedded and mobile access devices have power, data storage, display, and input limitations

Use virtualization and mediation in combination with abstraction to provide support (Section 3.2.1) Discover, assemble, invoke, and maintain services (Section 3.3) Mobility support for: routing to mobile receivers; to locate mobile access nodes (Section 4.2) Design to function autonomously and in intermittent offline mode (Section 3.2.3) Adapt the operation to the context of limited ICT resources (Section 4.3.3), limited local energy self-sufficiency ( Section 4.3.4 )

Interaction with external system ICT context aware User context aware (personalized)

Aware of the physical context

Awareness of local resource availability Volatile remote resource availability Smart devices are often personalized, but users can interact with many other impersonal services. User context is dynamic: it varies with activity, time, etc. The devices are active and in a relatively passive physical environment.

Operating System vs. App Compatibility Devices act as a source of personal space that can be extended to other devices. Support for dynamic profiles of direct and indirect users (section 5.6) Limited knowledge of the physical world (section 7)

Introduction

From the manufacture of micro-sized devices, backed by a powerful service infrastructure, other smart devices such as pens, ID cards, calculators, cards, glasses, paper, and watches will emerge. Thus, when looking at the Personal Reminders application (Section 1.1.2.1), features such as contextual audio and video (AV) capture, playback, and annotation can be accessed through a variety of devices, such as digital cameras, phones cell phones, games consoles, key rings, pins, etc., since the electronics of the device is getting smaller and smaller. These devices must be easy to use and self-configuring to discover the relevant network and services. Mobile users can dynamically discover displays and printers in the environment, and users can choose which ones to interact with based on their personal preferences. You can mix and match different resources from different providers. It is worth briefly reviewing the difference between services and devices. Devices represent an execution environment for (service) processes, which includes a limited and device-specific set of local ICT system resources (for the device) and resources from the physical environment. Device models cannot be assumed to have a ubiquitous virtual computing environment to enable access to services and ubiquitous execution on any device, as device access to the virtual computing environment may be volatile and non-interoperable. In Chapter 1, devices are called virtual embedded computing systems.

3.1.3 Distributed system views Smart devices embed user access to distributed system components, such as task-based information and services, e.g. B. management and control of resources, within a user-centered access device to a distributed ICT service. Distributed ICT systems can be modeled from several complementary perspectives2 in relation to different stakeholders in the system, such as network infrastructure providers, computing device and service infrastructure providers, individual users, and business users (see Figure 3.1). These views can be seen as architectural patterns, conceptual models that capture the essential elements of an ICT system architecture and their interrelationships. The system provides information or tasks from the perspective of the (individual) user. From the perspective of a business user, user access is controlled through roles and organizational policies. From the point of view of the information system, service or computer platform, the system is divided into three3 interdependent components that can be distributed and interconnected through a fourth communication component. These three components are: one or more managed information resources such as stored data and information resources such as sensors and databases; information manipulation processes; User interfaces for interacting with the resources and processes of a system. The information point of view takes an abstract view of communication, considering the network and resources as passive components and users and processing as active components. The vision of network providers on ICT systems defines two types of nodes, computing or service nodes and network or communication nodes, special purpose computing nodes that allow computing nodes to share the use of physical network connections ( see Chapter 11). The focus in this chapter is on the service view. The user,

The Open Distributed Processing Reference Model (RM-ODP) architecture (ISO/IEC 10746-3, 1996) defines views for business users, computing, information (distributed, service), engineering, and technology. The IEEE 1471 architectural model (ISO/IEC 42010, 2000) is a more general architectural model for software-intensive systems that is less specific in terms of defining viewpoints than the earlier RM-ODP model. 3 Different models of ICT systems have a different number of components. The term ICT (Information Communication Technology) is very widespread today, Garlan and Shaw (1993) define two components: information or computing and communication. Perry and Wolfe (1992) and Fielding (2000) define three general system components: processing, connectors, and data components. Alonso et al. (2004) define components for data-intensive systems such as resource management (for example, data storage), business logic (data processing), and presentation logic (data access).

Smart devices and services

User view information (I, C) (I, C) information

Red Ver Red (C) (C) Red

Service/Platform View User Interface User Interface

Tasks(P) (P) Tasks

Computer(P,I) (P,I) Computer

company view

Communications Middleware Services and Other Middleware Service

Benutzer (U) (U) Benutzer

Service Service Processes Processes

Resource Management Resource Management

Policies (I) (I) User Policies (U) (U) User

The role(I) (I) The role

Services (I,P)

Information Sensor Controls

Figure 3.1 Different views of the components of the distributed ICT system. The components are user access (U),1 service processing (P),2 communication (C3), and resources (R)4 such as information (I), sensors, controls, and other device hardware. Notes: 1 The user interface (UI) is also known as the presentation layer in business information models. 2 The processing or computation is known as the application logic or business logic layer in business information models. 3 Note C refers to communication in the acronym ICT and to computers in the acronym HCI, CPI, etc. 4 Resources, data sources, data storage, sensors, controllers, etc. they are encapsulated in the resource management layer in enterprise information models. Resources are hardware-bound, passive, and activated by users and services.

The business viewpoints and the network viewpoints are hidden through the service viewpoint and are covered in later chapters. In a monolithic service model, services, user access, and resources are all on one device or platform, whereas in a distributed service model, they can be distributed. The distributed service model has several advantages. Supports apps and stores that are distributed by nature, e.g. B. A company with several offices, factories and employees in different locations, which can be supported by messaging services such as email and information and knowledge portals. Distributed access increases the use of services that would only be accessed locally. Expensive resources can be shared. Better performance can be achieved by separating the computations so that they can be executed in multiple processes in parallel. Reliability can be increased by replicating tasks and data across multiple compute nodes, so if one set fails another copy can be configured to access. Some services are best distributed among collections of cooperating specialized services, although this is not necessary to provide flexibility for maintenance so that parts can be changed without disrupting the operation of the entire system.

3.1.4 Abstraction vs. Virtualization Systems architectures focus on the idea of reducing complexity both through a separation of concerns through modularization and through abstraction. Meyer (1998) uses five criteria to characterize the modules or components of the system: (1) decomposition capacity, to divide a system into modules; (2) composability to assemble certain existing modules to form a new system; (3) understandability of each separate module to make it easier for users to modify them and build a system out of them; (4) continuity, so that changes in one module reduce side effects in other modules; and (5) protection for failures within a module to attempt to limit effects within that module. To support the characteristics of distributed systems, modules must support high cohesion, perform well-defined functions, loose coupling, and few side effects. Pressman (1997) describes a series of

Introduction

Module cohesion from random (low) to functional (high) and a range of module docking from no direct docking to content docking. The second key concept to reduce complexity is to use abstractions that define the things that are important in a system and hide or make transparent the unimportant properties. Important types of transparency for distributed services include the following. Access transparency specifies resources that can be accessed from anywhere. Users only define what resources they need, not where the resources are located. Concurrency transparency makes it easy for multiple users or (application) programs to work on shared data without interference between users. Fault tolerance allows systems to mask partial system failures and increase availability. Replication transparency allows users or programs to be unaware that a system is using multiple instances of resources to increase reliability or performance. Migration transparency allows resources to move during use. Scaling transparency makes it easy to dynamically provision resources, allowing you to expand (or shrink) to meet demand. In practice, the ideal transparency of a single image for all resources at all times and under all conditions is difficult to achieve and is typically only achieved when the distributed system is operating normally. The use of standardized, homogeneous service interfaces facilitates interoperability. The use of standard homogeneous service implementations simplifies system administration. In practice, there are heterogeneous service implementations for the same service interfaces, produced by different service providers. The benefit to consumers is that they have more choice and perceive a greater critical mass of products when they are compatible. Service interfaces or implementations, or both, can be standardized. The availability of standard interface specifications does not necessarily guarantee interoperability. Specifications may be interpreted differently by different vendors, e.g. For example, parameters can have different default values or be used as an option instead of required. Providers may find it beneficial to extend standards, but they want to do it faster by going it alone rather than going through a standardization process that is often lengthy, so compliance with standard service specifications may vary. Vendors often advertise the benefits of upgrading to the latest version of their products because they offer enhanced functionality. The latest version may be less stable and cause additional expenses and maintenance costs, so users can often delay updates. Special versions and variations of the service may be required, e.g. B. Small portable screens are used for personalized mobile use, as opposed to large screens, which are used more for remote viewing by user groups. Abstraction alone often focuses on simplifying access to resources, e.g. B. The operating system and application services abstract away the details of disk addressing, eg. B. move a disk head across tracks and sectors of multiple spinning disks, into files and records with random access (Figure 3.2). Abstractions are defined by a well-defined interface4, e.g. a file system and a database system that hides the details of lower-level disk access. Abstractions alone do not necessarily support interoperability between different types of related interface instances, e.g. B. Resources designed to support one interface must be refactored to support another related interface, resulting in a lack of interoperability. Virtualization offers a way to solve this abstraction limitation: it supports the ability to map components on an interface at a given abstraction level to different interfaces and different resources at different abstraction levels (Smith and Nair, 2005).5 Virtualization is special

An abstraction that is a simplification of the functionality that an entity exposes from itself to the outside is called an interface. 5 virtualization technologies hide the physical characteristics of computing resources from users, e.g. B. to allow applications or the operating system of one host system to run under another. There are three requirements (Popek and Goldberg, 1974): (1) equivalence, so that an application running under the virtual host is compared to the native host; (2) resource control so that the virtual host has complete control over the virtualized resources; and (3) efficiency: a statistically dominant proportion of the machine's instructions must be executed without intervention from the virtual host.

Smart devices and services

web browser

access

file applications, e.g. B. Word processing, email, etc.

database applications

processing

SQL Commands DBMS Data as Records in Files File Manager Data as Pages (minimum data block on disk)

Resource management

Disk Manager Data as parts of magnetic disk tracks and sector information system resources

Figure 3.2 Abstract view of user access to file and database applications, who in turn see an abstract view of file system resource managers, who in turn see an abstract view of the data storage system on disk

useful for ubiquitous systems as it allows the use of different components in multiple services and at multiple levels of abstraction. A second key difference between abstraction and virtualization is that virtualization is not necessarily intended to hide and simplify all the details of access to services, although it can. The combination of virtualization and public interface specification characterizes an open system and makes it easy to add, modify, and remove system components. The important applications of virtualization are operating systems and virtual machines (Section 3.4). If resources and services are replicated in some way in a distributed system, the services can be designed so that if access to one or more specific components fails, alternate instances of them can be accessed instead. Any component for which at least one alternative is offered is called a redundant component. Components can be redundant because independent service providers adhere to the same interface specifications and offer similar or the same services. Components can be redundant by design. This is particularly necessary for critical components like network paths and directory services. Additionally, distributed systems could be designed in such a way that after a faulty component is detected, the system could attempt to reboot it and return it to normal operation, or automatically switch to use an alternate component.

3.2 Service architecture models 3.2.1 Partitioning and distribution of service components There are a number of schemes for partitioning and distributing services, depending on: (1) the application; (2) the type of communication service; and (3) the type of access device used. Figure 3.3 shows two different ways to design a chess app to run on smart devices. On a resource-intensive device, the app can run locally without using the network. On low-resource devices, a chess app runs remotely because there isn't enough CPU power to run the app locally. Devices with relatively high resource access can act autonomously and work in offline mode as monolithic or standalone devices - the device model. Relatively resource-poor access devices,

service architecture models

Low

Use of the application network.

Monolithic application: runs in disconnected mode on a device with high access to resources, e.g. B. calendar or chess apps

Hoch

Application CPU usage

A locally running application requires high CPU utilization, e.g. B. to calculate the next move in a chess application

High Device with low access to resources requires sustained use of the network to access remote services, e.g. B. Online games, voice calls

Low Low Resource access device used for presentation to view the game, e.g. B. Access a remote server to calculate the next move

Figure 3.3: The tradeoff between the use of local processing and the amount of communication required depends on the application and its design

Like lightweight mobile devices, they are often designed in such a way that most of the service execution is done over the network, the utility model. However, the need to deal with underperforming and unreliable networks, as well as the need to match power consumption with the available power reserves of mobile applications, demands a certain degree of autonomy and support of local processing, elements of a device model. There is a need to balance the level of local information storage and local processing with the level of remote processing and required communication bandwidth. Depending on the type of ICT infrastructure available and the type of applications, the balance may need to change dynamically. In the middleware service model, generic services such as communication and data management are offloaded from specific application servers and hosted on compute nodes so that they can be shared by multiple application services (Section 3.2.3). Service Oriented Computing (SOC), p. B. instantiated as an XML Web Service (WS)-based SOC, defines an explicit term of service (Section 3.2.4). An important variant of the client-server model is the mobile client model, where the network addresses of client devices can change during a service invocation session and service access is optimized for resource-poor mobile devices. The peer-to-peer6 service model supports more flexible service availability over more dynamic service infrastructures, such as service access over mobile ad hoc networks or MANET (Section 11.6.5.3).

3.2.2 Multi-tier customer service models Figure 3.4 indicates five different designs for UbiCom information resource systems based on how their A, P, and I components are distributed. These functions can be distributed across multiple nodes or different computer levels. From top to bottom and left to right, the layouts shown in Figure 3.4 look like this. In a monolithic, single-tier system, the entire application service resides locally when operational. The system is networkable so that, under special conditions, you can connect for help if your operations are interrupted or due to local outages. In a two-tier thin client server, the access device (or client or terminal device) supports access or presentation of data, service processes are executed remotely, and information associated with those services is stored remotely. remote. In a two-tier fat client server model, the access device can support some local processes and service consumption, but can also invoke remote services.

SOC is also known as the P2P computing model, but most SOC designs appear to be more accurately characterized by a client-server model.

Smart devices and services

a C

A CPI?

Client

personal computer

skinny client

monolithic

a C

application

yo c

Data

a personal computer

YoPC

Fetter Kunde

Server

a C

personal computer

thin client application

personal computer

Server

YoPC

Banco

in content C

Figure 3.4 Different designs for sharing and distributing information (I), processing (P) and access to services (A) using communication (C). Note: The numbers on the arrow indicate the order of interaction.

In multi-tier systems (3, 4...N tiers), instead of access devices directly connected to the end service nodes, different numbers of intermediate nodes can be used. Examples of the use of individual intermediate nodes that result in three-tier systems are designs that decouple service access from service delivery through the use of discovery services (Section 3.2.2), and services that act on behalf of of the access node to simplify operations (Section 3.2.2). 3.2.2.4). Often, application processing and application data are placed on separate nodes, resulting in a four-tier system. Various application support services and generic application support services can be defined, called middleware services (Section 3.2.3). A typical e-commerce application consists of four layers: a user requests a service from an application to download content, the application checks if it is ok with a banking and authentication service; if OK, the content is retrieved from the content server and sent to the access device. Application services can also be designed to be distributed over five or more tiers, depending on the application, e.g. B. Banking and authentication services could be separated, resulting in a five-tier system.

3.2.2.1 Distributed data storage Some of the components, such as (information) resources, processing, and access, can be further divided to support different types of applications. The types of systems in which information resources are partitioned and distributed (see Figure 3.5) include: Transactional monitors, in which data transactions are created from distributed data sources, such as Centralized analysis of data is supported centrally stored data subsets drawn periodically from distributed data resources; In distributed databases, where queries are distributed across multiple heterogeneous databases, each individual database is mapped to a common form by a database container.

3.2.2.2 Distributed Processing Although a single CPU server-client architecture is the dominant processing model used in distributed systems, sometimes more processing power is needed for short periods of time. One way to achieve this is to partition the processing, distribute it among multiple remote processors, execute each part of the processing in parallel, and then reassemble the results of each part to form the whole. For this to be worthwhile, the time gained by the increased processing needs to be significantly greater than the time spent dividing and distributing the tasks, collecting the individual results, and putting them back together. There are several different computer architectures for achieving parallel processing. Previously, support for parallel processing was only found on supercomputers that have hundreds to tens of thousands of CPUs and independent memories, such as B. massively parallel and specialized computers.

service architecture models

transaction processing

Access1 (A) The transaction

Access2 (A) The transaction

Monitor: route, schedule, monitor

data warehouse

distributed database

Access 1 (A)

Access n (A)

Data Mining (P)

OLAP (P)

Jobs P

metadata repository

Aggregate Data Transaction Manager: Record Lock, Checkpoint, Log

lock records

Access 1 (A)

Distributed Query (P)

Packaging (P)

R Refresh, Load, Clean, Transform Export

export R

Database Rules 1

Access n (A)

Export database R 1

export R

database 1

database

database 1

Figure 3.5 Information resources (R) can be divided within an information system

computer cluster. These have been used to perform high performance processing applications such as weather forecasting (Hord, 1999). Today, limited but significant parallel processing using general-purpose computers can be achieved by connecting general-purpose computers in clusters, such as B. Computer networks. Various processing capabilities can be incorporated into general purpose computers, e.g. a multicore processor containing multiple CPUs on a single IC chip (Kumar et al., 2003). It has the advantage of reduced latency and lower power consumption compared to the networked single-core CPU model. Other models for supporting massively parallel computing included the use of P2P computing (Section 3.2.6) and cellular computing (Section 10.6.1).

3.2.2.3 Client-server design The client-server model has the advantage of more centralized control over the distribution, but it has the disadvantage that the distribution and configuration of the servers is fixed. The client-server model is an asymmetric distributed computational model in terms of resources and direction of interaction. Servers are often very resource intensive, e.g. B. Larger storage capacity, more powerful CPUs to support intensive processing tasks, high bandwidth, always network connection to handle multiple service requests, and acts as shared data storage. Unlike servers, clients are relatively resource poor. Client-server interaction is also asymmetric: client processes on access devices initiate the interaction by making requests to application service processes on servers that are waiting for client requests. This asymmetry simplifies synchronization between clients starting to make requests while servers start waiting for client requests. The configuration of the system in terms of partitioning and distribution of service components depends on: (1) the available ICT infrastructure, such as the performance and reliability of network connections; (2) the resource limitations of the local device; (3) the remote services available; and (4) the type of application and service maintenance model. The relative scarcity of resources of some types of computing nodes, such as mobile devices, speaks to trust

Smart devices and services

on external servers, as long as the communication network supports on-demand remote service access: a thin-client server model. An example of a thin client server model is a mobile device that only supports access to the service through a mini web browser. All processing on behalf of clients, see Section 3.2.2.4, is done on the server, requiring significant server resources to handle many (possibly heterogeneous) clients. A thin client server model is often considered easier to maintain because maintenance can be performed remotely at a centralized server location, rather than having to be performed at each distributed local access node (client). However, web browser-based thin clients support a very limited application platform in terms of support for protocols other than HTTP, support for rich and interactive user interfaces, or support for sophisticated application logic. The need to deal with unreliable and underperforming networks dictates a degree of autonomy and use of local processing and data resources, i. h using a heavyweight client-server model. The design of the interaction to tolerate intermittent network access is discussed in more detail in Section 3.3.3.9. The thick client model is appropriate when the access device has a higher system specification than thin client devices and when network connectivity is limited. It has the advantage that some of the processing and storage is offloaded from the server. The type of processing required in the access device depends on the type of application. Location-based services can determine the location of the access device and convert its location coordinates into a form that can be used to interoperate with other applications, such as mapping applications, thus avoiding remote transmission of data to be transformed. . Access devices, such as point-of-sale terminals, can be used to collect data locally, do some local processing, and then upload it later.

3.2.2.4 Proxy-based service access Some applications use a client proxy to hide the complexity of the communication so that it is not compatible with the client access process. Proxy-based service access can:

• Offload presentation processing and network processing from low-income customer access device

• • •

• •

to the third proxy node, e.g. B. the client proxy could pass XML to a web server but use a simpler messaging protocol when communicating with the mobile device, avoiding the need to parse complex XML data structures in the access terminal; hide the heterogeneity of different types of terminals and different types of networks from access applications; simplify and add access to multiple service providers. A proxy can request a default connection, which reduces processing and communication with wireless end devices. You can support a mobile portal model by adding content from multiple service providers. Reduce the complexity of the communication used in the access devices, e.g. For example, complex hierarchical data structures often need to be encoded and decoded into a more efficient serialized data structure for streaming (see Figure 3.6), p. by compressing data or transcoding data on the server side, reducing the amount of wireless bandwidth consumed (Chapter 11); Allow devices to work in a disconnected state intermittently. Devices may turn off, go out of range of the wireless network, or simply choose to operate in a disconnected mode (Section 3.3.3.9); Protect network-based applications from the mobility of access devices (Section 4.2).

Mobile apps on resource-constrained access devices tend to use at least a three-tier design model. A thin client is supported on the access device node, application services reside on another node, and any mediator or proxy clients reside on another node.

service architecture models

(Video) How to Hack Webcam, Microphone & get Mobile Location using a Link

object6

object5

Object4

object 1

object2

Client

object6

object5

Object4

Encode/decode data for Object3 streaming

Attorney

object 1

object2

Object3

Server

user query

Result Optional confirmation of the received result

Figure 3.6 Using Proxies to Simplify Network Access by Transparently Encrypting and Decrypting Data Transmitted on Behalf of Clients and Servers

Using proxies in this way overcomes the problems associated with limited network coverage and can support reliable communication. The proxy client also has several drawbacks that must be considered when designing the proxy client. Disadvantages include: Because it is a single point of failure, using an additional network hop can increase latency. In some cases, proxy clients can be located far from application web servers and access devices, further increasing latency. Some of these problems can be addressed by replication of distributed proxy servers located at optimal points in the network.

3.2.3 Middleware The diversity and heterogeneity of service access increases the complexity of application design to access this expanded range of services. Therefore, middleware was introduced between applications and the operating system to allow applications to hide and simplify access to the heterogeneous and distributed resources of multiple networked computer systems (Bernstein, 1996). The middleware basically breaks down a set of generic services, e.g. B. access to the database, access to the file system, messaging, time service, directory service, etc., from the application services and from the operating system, so that they can be independent of the application and the OS. In theory, this makes the operating system (OS) much more compact and flexible. There is no need to rebuild or reboot the operating system every time new types of hardware are added. The middleware itself is distributed, but this is transparent to the application. The middleware model also makes it easier to define applications because the middleware can handle the complexity of handling the detailed use of system services. However, it can sometimes be useful for some applications to have some awareness of lower-level interactions and not completely hide access to resources from the middleware (Figure 3.7). For example, application awareness can better handle message latency, service activation, and component replacement that require synchronization to avoid deadlocks (when two or more processes are waiting for another to release a resource) and deadlocks (when two or more processes are waiting for another to release a resource). processes are constantly changing. cope with others, there is no progress). Alonso et al. (2004) consider (heterogeneous) model systems such as EAI (Enterprise Application Integration) and SOA (Service Oriented Architecture) to model enterprise supply chains as independent.

Smart devices and services

Application awareness of ICT context The full application provides a complete interface of the ICT system, no middleware is used

Partial middleware handles part of the complexity of interfacing with the IT system

No middleware hides the complexity of the ICT system from the application

Figure 3.7 The trade-off in the use of middleware to hide the complexity of ICT system access from middleware applications and service types

of middleware models. This is because each type of system has its own type of middleware. By contrast, systems of systems require heterogeneous middleware to work together. This requires significant extensions to the middleware model, as these types of system interaction schemes may require harmony between different policies, such as security policies, requirements, and data formats. This requires more sophisticated methods for modeling services and service interactions (Section 3.2.4).

3.2.4 Service Oriented Computing (SOC) Service Oriented Computing (SOC) or Service Oriented Architecture (SOA) focuses on services such as computing or information processing components that are autonomous, heterogeneous, run on different platforms and possibly owned by different organizations. XML-based web services and computer networks are common examples of SOA. The OASIS standards forum centers its SOA7 model on the concept of ownership: an SOA is "a paradigm for organizing and using distributed capabilities that may be under the control of different domains of ownership." Its reference model defines the basics of visibility, service description, execution context, real-world impact, interaction, contract, and policy. Each of these is defined in terms of a more detailed conceptualization. Other bodies, such as the Open Group SOA Working Group8, which defines SOA as an IT architectural style that provides business agility and "unlimited information flow", and the OMG.9 are also taking similar initiatives to specify standard SOA models. service concept can be characterized as follows:

• Descriptions (or specification) of a task (a set of one or more actions) offered by providers • • •

to user. Descriptions are supposed to be found. One complication is that the provider and the user may not have a common understanding of the specification or how to specify a service. Results: Service is the means to achieve a defined result for a task, e.g. B. A repair service allows you to resume normal operations within a specified period of time. Offers: (or offers) to perform a task on behalf of another. An offer assumes that the provider is available. Competition: To fulfill the task, a provider can e.g. B. Publish ratings that have been validated by an independent regulatory body.

OASIS SOA RM (2006) OASIS Reference Model for Service Oriented Architecture V 1.0. Available at http://www.oasis-open.org/committees/. Accessed August 2007. 8 Open Group (2007) The Open Group SOA home page. http://www.opengroup.org/projects/soa/ Accessed August 2007. 9 OMG SOA (2007) OMG SOA Special Interest Group home page. Available at http://soa.omg.org/. Accessed August 2007.

service architecture models

• Execution: The actual execution of the service on behalf of another person. • Composition: Multiple services may need to be composed before they can be executed in terms of a result and time constraints.

• Restrictions or Policies – For a service that can be configured by the user, eg. B. for a taxi service “don't drive too fast”, or from the provider “don't exceed the speed limit”. Service design may not explicitly define all functionality. In the simple and informal case, the service description can only be defined from the provider's point of view. All other features are implicitly assumed by virtue of an informal agreement. For business services, it is often necessary to explicitly define each of these functions. Some service design models may distinguish between information processing tasks and information retrieval tasks. An SOA is an architectural paradigm that encourages the development of potentially ad hoc applications from a set of loosely coupled, independent, and mutually collaborating services. SOA services can be divided into three functional layers: basic (lower), composite (middle) and management (upper) (Papazoglou et al., 2007). These functions are described below:

• Service Discovery (Basic Function): Service descriptions are discovered, published, and displayed • • • •

Use metadata that can be advertised and discovered through third-party intermediary services, such as directory services (Section 3.3.2); Enterprise Service Bus (Basic Feature): Supports service, message, and event-based interactions with appropriate service levels and manageability; Service invocation (basic function): Services are invoked through public interfaces over an open service infrastructure. Services are accessible over a public network and are defined using standards-based representations (Section 3.3.3); Service Composition: Services can be combined from simpler components to more complex, compound, executable service processes at runtime (Section 3.3.3.9); Service management: Services are managed by third-party policies between the user and the provider by exchanging contracts based on schemas and service level agreements (SLAs), e.g. B. Dan et al. (2004).

The SOC is a design model that is independent of any particular technology, e.g. B. web services or events. This independence can be achieved by limiting the number of implementation constraints at an abstraction level at the service interface. SOC only requires that functions or services be explicitly defined by a service description language, e.g. using WSDL, and have interfaces to use these descriptions to perform useful business processes (Papazoglou et al., 2007). Not all of the above service features and not all SOC features may be supported by all SOC implementations, e.g. B. Consumer device-based SOCs (Section 4.5) tend not to define service offerings, competition, and outcome, and service composition and management are often fairly simple and avoid politics and contracts.

3.2.5 Grid computing Grid computing refers to distributed systems that enable large-scale coordinated use and sharing of geographically distributed resources10 based on standards-based persistent service infrastructures, often with a high-performance orientation (Foster et al., 2001). network computing

Requests for the use and sharing of resources, such as computing resources and computer memory, are called jobs.

Smart devices and services

it specifies standards for high performance computing infrastructure rather than support for fault tolerance and support for highly dynamic ad hoc interactions, which are more the focus of P2P systems (Foster and Iamnitchi, 2003). In practice, three main types of grid systems occur: (1) computational grids, which have a higher aggregate computational capacity available for individual applications than the capacity of any constituent machine in the system; (2) data grids, which provide an infrastructure for synthesizing new information from data stores such as digital libraries or data stores distributed over a wide area network; and (3) service networks, which provide services that are not provided by a single machine. This category is further divided into multimedia, collaborative, and on-demand grid systems (Krauter et al., 2002). Each of these grid types currently uses a web-based SOA model. Much effort has gone into developing open models and specifications for grid computing.11 Grids typically focus on providing a single virtual computer view of distributed systems made up of fat servers and bulky client computers that have more likely to share high-bandwidth, high-availability fixed networks than thin client devices that can connect over more volatile, low-bandwidth networks, including wireless networks. The resource model used in the network focuses on sharing computing and data storage resources. By contrast, the resource models used at UbiCom focus on a broader range of ICT and non-ICT resources, including energy and environmental controls.

3.2.6 Peer-to-Peer Systems Peer-to-Peer, or P2P, systems are service infrastructures.12 They can be defined as distributed systems made up of interconnected nodes capable of organizing themselves in network topologies to share resources such as content, Cycles of CPU, memory and bandwidth, capable of adapting to failures and accommodating transient populations of nodes while maintaining acceptable connectivity and performance without the need for mediation or support from global centralized authorities or servers (Androutsellis-Theotokis and Spinellis, 2004). Instead of concentrating sophisticated server processing and resource management on a relatively small number of nodes or specialized servers, service access devices can also act as on-demand servers for some resource-intensive customers. Computer nodes in a P2P system can act as clients and servers. Rather, client and server are viewed as dynamic peer-to-peer organizational roles that can be changed as needed. P2P interaction often uses an ad hoc application router or service overlay network (Section 11.7.8.4). Ad-hoc generally applies to physical networks where communication takes place without building pre-existing infrastructure between the communicating computers, e.g. B. Mobile ad hoc networks or MANET. P2P service infrastructures can be superimposed on ad hoc networks. A P2P service infrastructure appears to be a very suitable system design to support many of the smart device properties listed in Section 3.1.2. P2P applications include: content sharing, in which

11 See Open Grid Forum (OGF) at http://www.ogf.org/, accessed January 2008. Note, however, that others do not refer to this grid-specific OGF model, but rather in a much more flexible way Sense as an interconnected network (network) of distributed resources, such as sensor networks, e.g. B. Hingne et al. (2003). 12 From the beginning, the original Internet was designed to function as a P2P network (Chapter 11). Many applications of P2P models focus on a network model rather than a service infrastructure model, as computing nodes act as ad hoc communication middleware and message routers to distribute information in a distributed fashion. However, this message routing is still done at the application level as a "Service Oriented Network Overlay". The nodes behave as application layer message routers, service proxies, and service gateways rather than network layer routers, which is why P2P is referred to as a service rather than a network infrastructure.

service architecture models

everyone can share and post; spontaneous real-time user collaboration like VoIP; Ad hoc interaction of wireless devices, such as B. Home devices discovering each other and sharing traffic reports on an ad hoc car network as network nodes; Distributed computations, such as sharing computing power to solve complex problems, e.g.[Email protected]Project (Anderson et al., 2002), distributed database systems, sensor network applications (Section 6.3), and various middleware services such as data protection through anonymous distribution of information. A P2P computing model offers several key benefits:

• Lower operating costs for content sharing: eliminating the cost of specialized servers and spreading maintenance costs among multiple low-cost peers that can interact;

• Performance improvements: all node resources can be used for storage, computing, and data exchange instead of concentrating resources primarily on server-type nodes;

• Ad hoc resource utilization and sharing: When the demand for certain services peaks, more nodes can act as servers, such as file servers, to meet the demand;

• Autonomous ownership and control: peers can have more and do more • •

decentralized and autonomous control over your data and resources. It does not depend on a central server to collect and send information. Anonymity and privacy: Session-based IDs and addresses can be assigned and hidden and masked when data is forwarded; one cannot know who created the data, who consults it, who stores it, etc.; Fault tolerance: there are no central servers that can be attacked or cause a complete system failure; instead, alternate routes and servers can be used if one fails (Section 12.2.6).

The challenges in the design and use of P2P systems are:

• More complex coordination is often required. In contrast, in client-server interaction, there are clients

• •

Initiate content downloads and uploads while servers respond to clients to satisfy those requests, or notify clients when service updates are available. On the contrary, it is not so clear how to start sessions, e.g. two interacting peers can choose to wait or send each other. Nodes can act as parasites: Nodes may like to play the role of service requesters, but they are always configured to deny requests from others to use their resources. More complex security may be required, as identification can be masked, making access control more difficult. Network address discovery: Peers must have dynamically assigned network addresses and discover addresses of destination nodes. Using broadcasting to find out the network addresses of others can flood networks. Ad hoc network routes: Ad hoc routes must be created and discovered between nodes, inefficient multi-hop routes can be formed to connect source nodes with destination nodes. Service discovery: how to recognize selective nodes from which services can be invoked versus the inefficiency of distributing services to all nodes.

There are three main variants of the P2P system, depending on the type of computing nodes: pure, decentralized, P2P; partially decentralized P2P (supernode); and decentralized hybrid P2P (see Figure 3.8). These can be grouped into two main types of topologies for P2P systems that overlap.

In 1995, David Gedye suggested this.[Email protected], Search for Extraterrestrial Intelligence, a project to use a virtual supercomputer consisting of a large number of computers connected to the Internet to discover intelligent life beyond Earth. For users, this application acts as a screen saver that periodically downloads and processes astronomical data and notifies someone about detected patterns; see http://setiathome.berkeley.edu, accessed December 2007.

Smart devices and services

Look for a

Servers Nodes C

G F

One-hop access

new node

Seek

Centralized directory node (used for searches in Hybrid P2P)

link not called

P2P only

P2P hybrids

node with content

Supernudo C

GRAMS

F.J.

node with content

Partially P2P

L K

Figure 3.8 Three types of P2P systems, pure, hybrid and partially decentralized

the underlying physical network. Unstructured overlay networks, e.g. B. Ad hoc networks are independent of any physical network topology and use decentralized and partially decentralized nodes. Structured overlay networks depend on the physical network topology of the nodes and use hybrid distributed nodes (Chapter 11). Instead of connecting each node to some other node to form a network, the nodes only connect to their nearest neighbors. A purely P2P service infrastructure uses no notion of fixed clients or servers, just peer nodes with equal rights, simultaneously acting as dynamic servers and clients (called servers) for other nodes, depending on supply and demand. Service infrastructures like Gnutella or Freenet (Androutsellis-Theotokis and Spinellis, 2004) use a (purely) P2P organization for all purposes and are sometimes called true peer-to-peer networks. In partially decentralized P2P systems, not all nodes are created equal, some superpeers or supernodes are chosen to act as middleware servers, to act as network repeaters for other nodes in a VoIP application, or to store Cached indexes to distribute the content of a group of nodes to locate B. KaZaa and Skype VoIP (Montresor, 2004). Peers are automatically selected as supernodes if they have sufficient bandwidth and processing power. They are also fault tolerant in the sense that new supernodes can be chosen to replace old ones that have failed. In hybrid P2P networks, e.g. B. Napster, a client-server organization, is used for certain tasks and interactions, e.g. B. looking for services, and a P2P organization is used for others, e.g. B. calling services used. There are three basic content access processes in distributed systems: (1) to identify nodes; (2) to register nodes that provide content; and (3) to search and retrieve content. How these work depends on the types of nodes and the topologies to structure

Service delivery life cycle

the nodes (Milojicic et al., 2002). Hybrid P2P systems tend to use client-server interaction to register and search for content in a central directory and use P2P interaction to retrieve content. P2P systems tend to use a decentralized approach to search for unknown recipient nodes. They use message broadcasts or message floods to ensure that all parts of the system are searched. There are several challenges with message streams: They scale poorly as the size of the network increases. You can use resources and processing power on all nodes, including those that may not respond to the request. It consumes significant network bandwidth as it can traverse all nodes via all possible paths. Message loops and partitioned networks can cause transmissions to take too long or end prematurely. Overlay networks of nodes can be used to avoid loops and partitions in the underlying physical networks, and the time to live (TTL) of messages can be limited to make flooding more manageable. This still leaves the challenge of efficiently identifying and annotating nodes that contain particular services and avoiding having to route requests through irrelevant nodes and paths. A key solution to this challenge, used in both pure and partial P2P, is the Distributed Hash Table (DHT). Each node is assigned a random ID, and each peer also knows a specific number of neighboring peers. When a document is published (shared), it is assigned an ID based on a hash of the document's metadata, which summarizes its content and name. Each node then forwards the document to another node with the ID that most closely matches the ID of the document. This process is repeated until the next pair ID is the current pair ID. When a peer requests the document from the P2P system, the request is forwarded to the peer with the ID that most closely matches the ID of the document. This process is repeated until a copy of the document is found. The document is then transmitted back to the originator of the request, while each peer involved in routing retains a local copy (Milojicic et al., 2002).

3.2.7 Device Models Devices embed services into fixed systems to support task-specific functions. Devices tend to be very heterogeneous and ICT resources are restricted to varying degrees. In addition to MTOS devices (Section 3.3.3.9), devices include: embedded control devices (Section 6.5), mobile devices (Section 4.2), smart cards (Section 4.3), and microelectromechanical devices and sensors (Section 6.4). Unlike information-based systems, e.g. B. WS-SOC, UbiCom devices focus more on UbiCom Contextowareness (Chapter 7), iHCI (Chapter 5) and Autonomy (Chapter 10) system properties. They inherently support more automated network configuration for more dynamic, fragmented, volatile, and less structured networks. They support more automated service configuration of more dynamic and less fluid service and resource spaces. Specific designs for dynamic self-managing networks and service discovery are discussed in more detail in Sections 3.3.1 and 3.3.2.

3.3 Service delivery lifecycle The provision of application services to smart devices involves managing distributed services throughout their lifecycle (see Figure 3.9) and not just in specific phases, such as service discovery . For example, smart devices that operate in dynamic environments such as B. smart mobile devices, do not assume that services remain static during service operation. Service provision design has to deal with intermittent access to the service and handovers between different instances of the service, e.g. B. wireless communications handovers. This implies two separate aspects, first, the definition of a generic life cycle model for the provision of services, and second, the management of that life cycle. In a simple service delivery lifecycle model, only two of the five services are implemented

Smart devices and services

Announcement (P), Discovery (A) Selection (A), Configuration (A|P) Composition (A)

creation

Call (A) Coordinate (A,P)

Exit (A,P) Delete Records (P)

Performance

resolution

Maintenance Retirement (A), upgrade descriptions (P) Reconfiguration (A), upgrade composition (A) Figure 3.9 The service lifecycle: Intelligent services include operation and management throughout the entire lifecycle. P and A indicate that service processes and service access are active during each phase

While the components of the model are active, services processing or provision of services and access to services or clients, the other three components, communication, stored information and information sources, are treated as passive components. In the service creation phase, service processes are registered in service directories. Service requesters at access nodes look for services (information processes and repositories). Services are selected, configured and several services have to be combined, e.g. For example, multiple services must be assembled to capture, annotate, transmit, edit, store, configure, and print images. In the operational or service execution phase, services are invoked and multiple interconnected services may need to be coordinated. In the service maintenance phase, service processes, access settings, and service compositions can be updated. In the service resolution phase, services can be temporarily disconnected or canceled by the processes themselves or by the requesters. Services can also be canceled and permanently removed. The design life cycle depends on the application requirements, e.g. B. the type of mobility required (Chapter 1). For example, a static device, such as a desktop A/V receiver, can support both starting and running dynamic services. This allows the device to be preconfigured to factory defaults and then shipped for use in different regions where it must recognize and adjust to varying regional RF transmission signal sources. This can also allow a static smart device to switch to an alternative service provider if it fails, as long as the user has permission to access it, perhaps through a different service contract.

3.3.1 Network discovery In general, network discovery should precede service registration and service discovery.14 Network discovery is used by mobile nodes dynamically and when new nodes are introduced into a network. A domain name service or DNS is used to map an IP address to a network node name and vice versa. A common approach to network discovery is to use Dynamic Host Control Protocol (DHCP) to ask a DHCP server for an IP address to lease for a specific period of time. The lease allows a limited set of resources, in this case network addresses, to be periodically refreshed and reused by active nodes and released from inactive computing nodes. Some nodes offering long-term services

Discovery services can also be divided into white page search (name, address, etc.), yellow page search (searches by type and attribute), and green page search (information on how the service is invoked). .

Service delivery life cycle

Static IP addresses can be assigned to printers. The complexity of using DHCP lies in configuring and managing DHCP servers and detecting and resolving duplicate IP addresses because: multiple DHCP servers can issue overlapping addresses; permanent IP addresses may conflict with dynamically assigned ones; Inactive clients can try to use a newly assigned address. Zeroconf or Zero Configuration Networking is a set of techniques that automatically create a usable IP network without any configuration or dedicated servers. This allows novice users to connect computers, network printers, and other devices and expect them to work automatically. Without zeroconf or the like, an expert user must configure special services like DHCP and DNS, or manually configure each computer's network settings, which is a tedious and challenging task for the uninitiated. Zeroconf currently solves three tasks automatically: choosing network addresses, giving itself an address, discovering names, and discovering service addresses. Both IPv4 and IPv6 have standard methods for automatically selecting IP addresses. IPv4 uses the 169.254.any link-local address pool, see RFC 3927.15. For IPv6, Zeroconf can be used, see RFC 2462. There are two similar ways to find out which network node has a specific name. Apple's Multicast DNS (mDNS) allows a network device to select a domain name in a local namespace and advertise it using a specific multicast IP address. Microsoft's Link-Local Multicast Name Resolution (LLMNR) is less commonly used and has not conformed to IETF standards.

3.3.2 Service advertisement, discovery, selection and configuration When service providers and requesters are static, there is little need for dynamic service discovery. Dynamic service discovery is required to allow service requesters to switch providers when requestors or providers are mobile (Section 4.2), when network access is intermittent (Section 3.3.3.9), and when requestors or providers are inactive. Dynamic service discovery16 involves decoupling service provision from service requirements and supporting dynamic advertisements and dynamic discovery of service providers and service requesters. There are two main approaches to dynamic service announcements and discovery: push and pull. Push uses broadcasts or multicasts to notify a number of unknown parties, e.g. for example, Bluetooth, to advertise17 available service requirements or service capabilities18. Service request or service description broadcasts are a subtype of message broadcasts to unknown message recipients. The requester or supplier performs the comparison. Pull uses search to search or search through lists of requirements or skills that have been previously published in a directory maintained by a well-known third-party provider, e.g. B. Jini, UPnP, UDDI, etc. The third performs the comparison. The advantage of directories over broadcasts is that they minimize service discovery network traffic. The disadvantage of directories is that

15 The IETF Request for Comments (RFC) specifications are available at www.ietf.org/rfc.html, accessed August 2007. 16 The scope of the term service discovery varies depending on the specific design. It could simply mean requesting the list of available service providers that match a request. It may or may not include service selection, service configuration, service name for address resolution, and even service invocation. 17 Ads can be designed to occur at specific times: regularly, regardless of whether there is an audience or not; only if there is any hearing available; only when a specific audience type is recognized: multicast versus streaming. 18 Note Researchers often consider service capabilities (provider) to be synonymous with service promotion and service description, but others, SOA, differentiate capabilities related to competency to provide service from service description.

Smart devices and services

this requires that the directory be maintained by a third party, the directory must be available, and the directory must have a known location for clients and servers to find. There are several design dimensions to the service selection specification: requirements-based vs. capability-based vs. goal-based, exact vs. imprecise, and syntactic vs. semantic. It's more common to match requests from a single user to service descriptions from multiple providers in directories. The opposite of this is using Blackboard, a third-party service where service requests are posted instead of SOWs, and service providers search for service requesters to do business with (see Figure 3.10). While discovery identifies the set of possible matches, selection determines the actual service match to use. Request-based service selection (and invocation) often implies that the services to fulfill the request exist in the (external) virtual environment of the ICT system and not internally within the system itself. This raises the question of how a requester of a service knows if he himself can provide the service or not (because he has the necessary resources, experience, etc. or not). Depending on the benefit or cost model (Section 8.3.5), in some cases it may be more advantageous to source services internally rather than externally, or vice versa. This also applies to business services and ICT services. The process of determining whether or not a service exists internally involves self-description, self-awareness, and reflection (Section 10.3). External service involves matching the specific requirements of one system with available service announcements from providers on another system. Providers are selected by the directory on behalf of the applicant, or all matches are given to the applicant and the applicant must decide on the selection of the provider. This involves retrieving the contact details of providers that match the service requirements in terms of preferences and restrictions, and making a selection from multiple service providers and then invoking the service (Figure 3.10). Service matching19 can be designed to support exact or imprecise wildcard matches and conditional matches. Goal-based service matching determines which of the available services can be used and how they can be assembled to achieve a user's goal.20 A particular selected service provider can be directly invoked as long as the requestor is configured to invoke the service provider. service. A variant of this is the Service Broker, which differs from the Registry

search services 2

I Services C

Customer C

See Also

Low Level Programming - PDFCOFFEE.COM

(RPC). RPCs are complex to design so that remote communication looks like local communication, e.g. B. Classification of parameters (Birrell and Nelson, 1984). Because of this, RPC has evolved towards an object-oriented style of Object Request Brokers, where components interact via remote method invocation (RMI) instead of RPC (Emmerich, 2000).

3.3.3.3 Reliable vs. Unreliable Communication Another design issue is considering the reliability of the network to deliver messages without loss or delay and in the correct order. Accessing services over wireless networks is often less reliable than wired networks. Applications cannot guarantee network delivery and must detect and deal with message corruption and loss. Message corruption can be dealt with by various message integrity checks or at the network layer (Chapter 11). At the application level, a message log can use additional acknowledgment event messages to detect any delayed or lost messages sent and received. To handle lost messages, senders and receivers can know the states (stateful), keep some intermediate states about sent messages, or buffer sent messages. This way, replacement messages don't have to be created from scratch, which is a huge overhead for data that requires significant computation to generate. Message senders that do not maintain the state of sent messages are called stateless. To some extent, stateful communication is more complex to synchronize than stateless communication, as intermediate states may need to be compared for equivalence. However, this can consume less ICT resources when generating complex messages, since they do not have to be generated completely from scratch. Another point before trying to send messages again is to consider the consequences of doing so. Messages that can be repeated at least once without side effects are called idempotent messages, e.g. B. pressing the call button of an elevator again because the response is not yet complete. Other messages may be non-idepotent, e.g. B. a message requesting a withdrawal of money from a bank account. The challenge in this case is the application's requirements that the action only needs to be performed once. Due to partial sender observability, the sender may not be able to distinguish between a sender crash before sending the message, the loss of a sent message, a remote server crash, and the loss of the received message. In case the receiver fails and the received message is lost, it's okay to retry the request, but not if a sent message is lost.

3.3.3.4 Caches, read-ahead, and write-behind Two design patterns for intermittent access to the server are read-ahead and write-behind (Figure 3.13). With read-ahead interaction, information is pre-cached on devices when the network is available. If the necessary data can be retrieved from the cache, it is called a cache hit. If the required data cannot be retrieved from the cache, this is known as a cache miss. In this case, the cache must send a request to get the data from the feed. include design decisions

Client (query, update) Read more

3. Consultation

client cache

Database server (Sales) 1. Query 2. Results

4. Delayed write results

1. Update

2. Update

Figure 3.13 Two design patterns for intermittent access to the server, read-ahead and write-delay

100

Smart devices and services

Decide what useful information to cache and how often to refresh the cache. Frequent caching results in fast site times, but removes the benefits of tax traffic reduction due to frequent updates. Less frequently cached data may be stale (out of date) and performance may suffer due to additional data operations to verify cache and sync. For lazy writes, updates are done in the local cache, while services are inaccessible and must be reintegrated later on reconnection. Concurrent local and remote updates may need to be synchronized. Write conflicts must be detected when the same data has been modified locally and remotely. Satyanarayanan (1996) discusses various techniques for handling cache misses and cache resynchronization. Model-based cache-missing design can take into account strong relationships between items and proactively cache highly allocated items with missing cache data. Optimistic replication allows replicated content to diverge in the cache at short notice to support concurrent work practices and to tolerate outages on poor-quality communication links. Changes can propagate in the background, conflicts can be discovered after the fact, and gradual agreements can be reached on the final content. Cache coherence can also be more usefully maintained at multiple levels of granularity.

3.3.3.5 Access to services on demand Access to services on demand implies that services are always available when needed. A common design for this type of interaction involves a pull-type request-response interaction. An entity initiates requests for application services over an always-on network connection using synchronous communication and waits for the response to the request (see Figure 3.12) from the server providers. For on-demand service interaction, a common design is based on a thin client stateless access node using synchronous communication over an always-on network connection. This greatly simplifies synchronization between presentation at the access node and server-based processing on the network. Often multiple request-responses are concatenated to complete an application session, eg clients must make multiple requests to prepare to invoke services, i. h Browse catalogs to discover services and select from multiple providers. E-commerce service invocation is a major example of on-demand service access, and often involves a sequence of multi-node service interactions through a retailer's customer-facing services, such as catalogs and sales, who acts as the customer's intermediary or representative for the customer's detailed support history. The back-end services of a retailer, such as inventory, delivery, and customer banking.28 Retail and e-commerce service interactions often use a thin-client access terminal that provides basic user interface controls for receiving a request for information and display simple response forms. In the simple case, this interaction uses synchronous communication and can be sequenced and controlled by the requester. This interaction can be grouped into two separate customer transactions: the item pick transaction and the item purchase transaction. The main design challenges of on-demand service access are ensuring that the right services are available when needed under the right conditions, and that the service access can tolerate volatile service access. There are several designs that help support access to ephemeral services. The interaction with the services can simply be repeated when the requests are not idempotent (Section 3.3.3.3). Local caches, read-ahead, and write-behind can be used to mask transient access to the service (Section 3.3.3.4). In addition, upon request, e.g.

28 Some of the details in the merchant part of the interaction are not shown here, e.g. B. the transfer of money from the customer's bank to the merchant's bank and the audit trail of receipts from the merchant's and customer's bank to the merchant or customer.

Service delivery life cycle

101

services fail, service detection and fault tolerance can be used to search for alternative existing service instances (Section 3.3.2), and service composition can be used to synthesize new service instances (Section 3.3.3.9) .

3.3.3.6 Event Driven Architectures (EDA) Gelernter and Carriero (1992) state that coordination mechanisms should be separated from calculation mechanisms. This supports several important benefits:

• Portability: providing coordination mechanisms independent of computational language; • Heterogeneity: Devices and applications are coordinated, even if these mechanisms are implemented on different hardware or language;

• Flexibility: Allows you to mix and match different coordination mechanisms and calculation mechanisms. Garlan and Shaw (1993) have identified and classified several different interaction mechanisms29 for distributed ICT systems. A common way to decouple coordination from computation is with an event-driven system, which supports loosely coupled control or coordination between event generators or producers and event sinks or consumers, e.g. B. Clicking buttons on a user interface (UI) generates events that trigger associated actions on services. This is also known as a publish and subscribe interaction. One or more nodes publish events while others subscribe to be notified30 when certain events occur (Eugster et al., 2003). An event is an input, such as a message or procedure call, that is of interest. An event can be significant because it can cause a significant change in state, e.g. B. a flat tire causes the driver of a vehicle to slow down. An event can cause a certain predefined threshold to be exceeded, e.g. For example, after driving a certain number of miles, a vehicle must be serviced to keep it roadworthy. An event can be time based, e.g. B. record a specific audio and video program at a specific time. External events can trigger services. The services can, in turn, trigger other internal events, e.g. B. The brake pads on the wheels are too worn and need to be replaced. Event-based architectures are an important interaction to support service-oriented architectures (Papazoglou et al., 2007). These are known as event-driven service-oriented architectures. In an event-based architecture, or EDA31, event consumers specify and register events that they want to be notified about with a special event dispatch engine that collects input and raw events and buffers them (see Figure 3.14 ). When event producers generate events, the event dispatcher collects them and distributes them to the event consumer that wants to be notified. The event consumer can perform its own check of whether the events meet certain constraints, and if they do, one or more actions or services are triggered. Event producers are said to be loosely coupled to event consumers. Event consumers can dynamically register and unsubscribe, e.g. For example, a customer can sign up to be notified of a new vehicle, but then unsubscribe when no longer interested. Event producers can choose not to produce new events. This loose coupling can lead to complexity and uncertainty. Event producers may not know who will respond and when they will finish consuming the event, e.g. b.

29 They categorized six main styles of interaction or coordination: pipe and filter, object-oriented data abstraction, event-driven, layered, repository, interpreter, and VM. 30 Event-based notification is similar to procedure callbacks in some respects. 31 Various standards consortia standardize event-based architectures, see http://soa.omg.org/SOA-docs/EDA-Standards.htm, accessed June 2007.

Smart devices and services

102

event dispatcher

Productor E2

Appropriate

Productor E4 Productor E3

consumer

source in/read

Outside

Those

E1-What

event loop

Producer E1

consumer E3

E4 E3

Registration

E2-What

Globo E2 E1 E3 E4

Tafel E3

consumer

E2 E1-filter

filter filter

Plot

consumer consumer

Figure 3.14 Shared repositories (left) and event-based interaction (right)

Users can get frustrated when they press buttons in a user interface, not sure if something happened or what progress was made unless feedback is provided. In some cases, recipients may not be given the full context or situation in which the event occurred to handle it effectively. Figure 3.14 shows a simple EDA. Multiple input events are entered into an event buffer, such as a first-in-first-out (FIFO) or first-in-last-out (FILO) buffer. Events are related to consumers who have previously expressed interest in those events. Pairing can be simple, e.g. B. An event is matched and sent to all registered consumers, or more complex. In a more complex design, events can be consumed using an ECA (Event Condition Action) paradigm; h Events are used to trigger actions when certain conditions are met, e.g. B. when the time equals T1, start the recording program on channel Y. These actions could, in turn, trigger new events. Events can be filtered in the event producer instead of a centralized event dispatcher shared by multiple event producers. Examples of the use of an EDA are asynchronous messaging systems (Section 3.3.3.1), sensor systems (Section 6.3) and reactive intelligent systems (Section 8.3.2). In a policy-based EDA, policies can be defined for action triggers, event buffering, and event logging. There are several design challenges with EDA, such as: B. dealing with event flooding and asynchronous coordination. In the event of a flood, some highly significant events may be lost in a large volume of insignificant events that cannot be processed in a timely manner. Event flooding can be handled by prioritizing events, allowing them to expire if not responded to within a specified time period, and using event filters. Applications may require event coordination when events can arrive in any order. EDA generally does not have persistence. There is no way to reuse current events. It's harder to keep things running in an error situation, recipients may also need to keep track of new senders to decide whether or not to subscribe. For efficiency reasons, most publish-subscribe systems broadcast events only when the recipients are recognized (Johanson and Fox, 2002).

Service delivery life cycle

103

3.3.3.7 Shared data repository In a repository style of interaction, two participants communicate by leaving messages with each other through a shared agent. Therefore, a shared repository system consists of two types of components: a central data structure that represents the current state, and a collection of independent components that operate on the central data store. There are two main subtypes of coordination, depending on whether transactions in an input stream trigger selection of processes to run, e.g. , a Blackboard repository. This represents and stores data created and used by other components. So we can talk about how data producers input data into the repository and data is pushed from the repository to data consumers. Thus, like EDA systems, repositories support independence and allow tolerance of volatile producers and consumers (see Figure 3.14). The main difference between EDA and Shared Data Repository is the persistent storage of input data and data management, e.g. B. consistency management. Examples of shared data repositories include electronic bulletin boards, knowledge-based whiteboard systems that include tuplespaces, and databases of relational data types. In addition to database vs. whiteboard, there are several other subtypes of repositories: passive vs. active, centralized vs. distributed, and caches. A repository can be passive: a sending component sends data to a selected repository component for storage, e.g. B. relational databases. The repository can be active: it stores and specifies its own actions triggered by certain states of the repository, e.g. B. Repository and rule engine, knowledge-based system. A repository does not have to be a single centralized repository, it can be highly distributed, e.g. B. the web. Repositories can also be replicated and distributed to improve availability. A cache is a repository for replicating the results of previous queries so that they can be reused by subsequent queries. An example of a Blackboard repository is a tuplespace in which tuples, ordered typed fields in which each field contains a value or is undefined, are stored in a persistent shared abstract space called a tuplespace. Three tuplespace operations are supported. 'Out' inserts a tuple into a tuplespace. "In" removes a tuple and "Read" copies a tuple from the space. In and Read match a tuple to a template tuple, using explicit values for some fields and placeholders for others. Of these, reading is the key operation that distinguishes it from event-driven programming, since read events are copied from memory and remain there for further processing.32 A separate component, a control unit, may be included. (not shown in Figure 3.14). to support read/write access by multiple simultaneous events, producers and consumers of data or knowledge. Johanson and Fox (2002) describe the use of a variation of the tuple space model called the event heap, which they applied in a distributed shared workspace application called iRoom (Section 2.2.2.3). Examples of tuplespace whiteboards that can be created using an underlying object-oriented even number layout include JavaSpaces33 and TSpaces (Wyckoff et al., 1998).

3.3.3.8 Enterprise Service Bus (ESB) model An Enterprise Service Bus or ESB34 supports messaging, web services integration, data transformation, and intelligent routing for SOCs; unties the provision of the service from the access to the service. This

"Really read" is a misnomer and should be more accurately referred to as "copy and save". This operation transforms a reactive intelligent system based on episodic environmental events into a model-based intelligent system that captures and accounts for sequential or past events in the environment (Section 8.2.3). 33 See Sun Microsystems Labs, JavaSpaces, available at http://www.sun.com/jini/specs/js.pdf. 34 Although the idea of the ESB was first reported around 2004, there is no current (mid-2008) standard specification for ESBs. Various designs and implementations exist, e.g. B.Keen et al. (2004).

104

Smart devices and services

The functions are distributed. There are two main types of designs: message-oriented ESB and service-oriented ESB. A message-oriented middleware (MOM) design for the ESB supports asynchronous messaging, transactions, publish-subscribe interaction styles, and routing at the application level (Section 3.3.3.1). However, MOMs may require a specific application layer or transport protocol, which may mean that gateways must be converted from an application that supports a different protocol. MOM itself should not be modeled as a direct web service or a first-class service35, but as an API. A service-oriented model for ESB, as opposed to a message-oriented model, provides broader support for three types of integration: (1) Integrate access to multiple services, e.g. B. Behavior as a portal; (2) integrate multiple application service processes, support workflows, mediation, and distribution; and (3) data translation support.

3.3.3.9 Volatile service calls Sometimes service access can be quite sporadic, eg. B. Wireless networks and mobile users tend to experience higher error rates and more frequent interruptions. This is due to: limited network area coverage and intermittent low-bandwidth access on some networks; network handovers when mobile users move between different base stations; Intermittent interference and variable signal reception. This can also be due to changes in heterogeneity in bandwidth and network coverage. In addition, access to services in open service infrastructures has to deal with variable access, as different numbers of requesters try to access a variable number of services that can be connected and disconnected and deal with the natural heterogeneity of user interaction. open services. Application design and middleware should take this into account; otherwise, the requests will be blocked or terminated and may need to be repeated and restarted. Basic designs for handling access to volatile services36 include the use of asynchronous communication (section 3.3.3.1), handling of unreliable communication (section 3.3.3.3), and message caching (section 3.3.3.4). Satyanarayanan (1996) summarized a more detailed set of design mechanisms for handling access to ephemeral services over potentially reliable low-bandwidth network links for applications involving simultaneous access to shared data. This parallelism can occur at two different levels: remote versus local, and write versus read. The design mechanisms for dealing with volatility are derived from his group's experience developing the Coda file system and the Odyssey mobile platform. Application customization support includes:

• Caching, logging, and synchronization: Updates that cannot be published immediately due to

• •

Disconnects can be pre-cached (read-ahead) or logged (write-delayed) for asynchronous exchanges. Additional mechanisms are needed to manage data consistency between cached data (temporary storage of) and more persistent data stores. adaptive transport protocols: for example, trickle-down mechanisms to propagate updates over low-bandwidth links37; Resource Reuse: The system can revoke the resources it owns and has temporarily delegated to other applications at any time. Resource reuse is essential in systems with few resources.

35 Modeling messaging itself as a complete service has advantages and disadvantages. The advantage is that it can be called by default and supports a generic service model. The downside is that this can be quite inefficient for high-performance systems, as it requires two service calls: a service call to invoke the messaging service and the messaging service call that transmits the message. 36 Volatile services are also sometimes referred to as occasionally connected or intermittently connected or ad hoc connected or spontaneously connected. 37 Low-bandwidth connectivity is also known as weak connectivity.

Service delivery life cycle

105

• Partial observability:38 access devices can detect (local) changes in their environment, eg. for example, the round-trip time of the message, to partially capture or estimate and then draw conclusions about the cause of these changes, for example. B. Network bandwidth limitations and respond appropriately.

3.3.4 Service Composition Composition is concerned with synthesizing new services and assembling more complex (composite) services from simple (atomic) services to achieve a user or application goal, and then running them collectively as composite service processes. Service processes are sequences of individual service actions that are scheduled to run. Service processes can include one or more entities, one or more actions, and one or more processes. Static organization of services and actions into expected default groups can cause requests to fail when a request is made to a service that does not exist or is not available. Dynamic service composition can be enabled to provide greater flexibility and encourage service access on demand (Section 3.3.3.4). Composition can be done incrementally over multiple rounds instead of a single round, and subsequent rounds can learn from the limitations and experience of running in earlier rounds. This is useful when it is not possible to plan the composition in advance, since it must be derived from the experience of running the service. Composition can be driven by a central entity (service orchestration) or by distributed peer-to-peer entities (service choreography).39 In general, the orchestrator tends to take a global view of service actions and constraints to represent participants. whereas in service choreography, participants are often more sensitive to local views of the service actions of themselves and the adjacent service processes of others. Service orchestration is easier to design than service choreography and seems to be used more frequently. Service composition can be specified manually or automatically using various service composition methods. Regarding the automation of service composition, different communities propose four approaches: business processes, workflow, semantic web and MAS planning. Business collaborations require long-term interactions driven by an explicit WS application process model, e.g. B. XML-based WS composition and execution standards from the business world, such as BPEL4WS, the Business Process Execution Language for Web Services (Bucchiarone and Gnesi, 2005) (van der Aalst et al., 2003). However, WS models do not inherently provide rich data structures and hierarchies like class-based hierarchies. They also require devices to have sufficient computing resources to understand and parse XML. The Semantic Web community focuses on reasoning about Web resources by explicitly stating their premises and effects using ontological models (Bucchiarone & Gnesi, 2005). Van der Aals and others. (2003) compared workflow management systems and web service composition languages using a number of patterns. The comparison shows that web service composition languages take over most of the functionality present in workflow systems. However, web service composition languages are more expressive than traditional workflow products.

Partial observability is also known as the global estimate from local observations. The term orchestration is used in music to refer to a conductor leading a group of individual classical musicians to play together. Choreography derives from the Greek words for "dance" and "write" and refers to the art of writing dance steps, allowing individual dancers some flexibility to provide their own interpretation of the dance steps. 39

106

Smart devices and services

3.3.4.1 Service interoperability The ability to communicate at the network level is not enough to interact at the service level. First of all, information and tasks must be coordinated, e.g. B. There is no point in sending if there is no recipient available. Second, a common syntax and semantics are needed to represent different information and tasks exchanged between heterogeneous systems. There is a difference between integrated services and interoperable (or federated) services. Service integration typically means that individual systems are statically connected, possibly combining data and code into a single complete repository, thus losing some of their autonomy. Service interoperability allows services to remain autonomous but dynamically connect to each other so that they can exchange data in a format that both understand. Distributed systems working together can exchange data in a variety of data formats, encodings, etc. Two main methods can be distinguished for the exchange of heterogeneous data: either a common or canonical data exchange format is used, or a receiver or sender creates the correct schema in which each party transforms the data in a way that the other can understand. . Fewer allocations need to be maintained when using a common interchange format compared to a sender or receiver scheme. Different expression levels can be used for data structures. Depending on the application, hierarchical structures such as XML, object-oriented hierarchies, and graph-based structures can be exchanged. Descriptions of syntactic or semantic structures can also be exchanged. Application layer communication protocols are specified as linear structures such as byte streams, but this means that hierarchical application data structures must be serialized for transmission and reassembled at the receiver. Various types of state information can be exchanged as metadata when one process invokes another. There are two main options: RPC or REST. With an RPC, information about the status of the sender is essentially transmitted to the receiver. However, the RPC is under the control of the recipient. In the Representation State Transfer (REST) model (Fielding, 2000), the recipient is seen as a set of resources identified by URLs, only representations of the resources are exchanged, not state information.

3.4 Virtual Machines and Operating Systems 3.4.1 Virtual Machines A virtual machine (VM) supports large-scale concurrent server execution by multiple users and enables cross-platform interoperability over a variety of hardware resources at multiple levels of abstraction. To understand the concept of a VM, the concept of a computer or a machine must first be seen from two different perspectives: from a process perspective and from a system (operating system) perspective (Smith and Nair, 2005). From the perspective of application processing, a computer consists of using processes that are held in memory in finite address spaces. Processes consist of a list of instructions defined in a high-level interface, the Application Programmer Interface (API), which are converted to binary digital instructions in a lower-level Application Binary Interface (ABI) in order to be executed. The underlying hardware, such as the CPU and I/O devices, is hidden by the virtualization software or VM Monitor (VMM), the API, and the ABI. The operating system view analyzes the details of how multiple processes can run concurrently on the hardware and when there are more processes available than hardware resources, unlike dedicated (embedded) task systems. From the perspective of the operating system, hardware interfaces, such as the instruction set architecture (ISA) and the system call interface, define the machine and act as VMM. The processes see the operating system as a virtual machine, while the operating system sees the hardware as a virtual machine.

Virtual machines and operating systems

107

Running portable code loads the virtual machine of the process

Application Development Environment APIs

Virtual machine

Language program at a higher level

virtual machine image

VM compiler

Interpreter

ABI portable code

hardware specific code

Figure 3.15: A HLL (high level language) program is compiled into intermediate code (portable code) (left). When this portable code is executed, it triggers the launch of a process VM (virtual machine) to interpret the instructions and convert them into executable code that runs on specific hardware (right).

A system virtual machine or system VM provides a complete persistent virtual environment for an operating system and an application process. The VM system was the original type of VM developed for mainframes in the 1960s and 1970s (Dickman, 1973). A system virtual machine allows multiple operating systems and applications to run on the same hardware. If one system fails, the others are isolated and continue to function. This is still a useful technique used in modern servers and server farms that need to support multiple users and applications and share hardware resources. A process or application virtual machine is a virtual platform that runs a single process. This virtual machine is created when the process starts running and only persists as long as the process is running. A high-level language processing VM performs actions specified in an instruction language (see Figure 3.15). Virtual machines are powerful because they can use a hardware-independent intermediate language, which supports the ability to run the same intermediate language on different sets of hardware and map them to heterogeneous hardware to actually run the program. The core component is an interpretation engine that translates a language statement on a per-statement basis into binary code as needed. This is in contrast to a compiler, which translates a program into binary code that can be executed on hardware in one step. An interpreter is a program that behaves like a CPU with a polling and execution cycle. To execute a program, the interpreter runs through a loop, repeatedly reading one statement at a time from the program, deciding how to execute that statement, and then executing the appropriate machine language instructions to accompany it (Smith and Nair, 2005). . Maintains a link between the state of the middleware being executed and the state of the binary code actually being executed. An intermediate language interpreter differs from a high-level language interpreter, such as a basic language interpreter or webscript, in that the output of the (native) machine code is optimized to give an interpretation of intermediate code performance comparable to the Comparable code execution. High level compiled language. For this, just-in-time or JIT compilation is used. This converts the code before executing it natively at runtime. The performance improvement over interpreters comes from caching the translation results of blocks of code, rather than simply re-evaluating each line or operand each time they are satisfied.

3.4.2 BIOS Often when a computer starts or boots, also known as booting, the software is loaded in stages. First, the BIOS or Basic Input/Output System, a type of firmware, is loaded. that takes getting used to

Smart devices and services

108

Load the operating system kernel. Firmware is a type of low-level software used to control hardware. It is a basic computer program that resides on special hardware, usually EEPROM ROM or ROM-like flash memory or read-only memory. It can be updated over the network through the operating system. The BIOS initializes various components and peripherals on the motherboard such as the CPU, system (primary) memory, graphics controller, secondary memory, I/O controllers such as keyboard, mouse, USB ports, and computer. system clock. Finally, the BIO loads and transfers control of the operating system (OS), and in modern computers, the operating system's own hardware drivers take control of the hardware from the BIOS. Also, many devices connected to MTOS computer systems are actually special purpose computers, e.g. B. Printers, these can also contain their own firmware in a ROM within the device itself.

3.4.3 Multitasking Operating Systems (MTOS) Desktop computers and smartphones (thick client devices) often require more complex multithreaded control and the use of a multitasking operating system (MTOS), also known as a purpose-built operating system. (GPOS) to support running multiple users and applications. The operating system can be viewed as a VM for processes (Section 3.4.1). The high-level model of an ICT system described in Section 3.2 consists of data input, data storage, data output, and data processing components that can be distributed and linked together through communication components. Generally, these components are virtualized by the operating system or OS for the processes. A special part of the operating system called the operating system kernel has priority control over the computer's hardware, such as the CPU, memory, and I/O ports for running processes. By design of the operating system, some core utilities are part of the kernel, others are outside. In addition, the operating system also supports software interfaces called device drivers to control hardware devices such as memory, displays, and input and output devices, and supports software for building applications. The operating system can be modeled as a layered architecture (Figure 3.16). The main functions of an operating system are to control access to data storage, such as memory and disk, to control data input and output, and to control data communication (Figure 3.17). computer

Application processes OS processes

monolithic core

micronucleus

operating system processes

Applications User interface Memory manager Network drivers

programmer department

E/A Manager

file manager

process manager

power manager

device driver

graphics controller

Hardware

Example of a bus schedule file system dialog,

Windows CE

RAM memory software interface RAM memory

Figure 3.16 The main components of an operating system. There are two basic types of operating system kernels: microkernel and monolithic kernel.

Virtual machines and operating systems

Begin

109

personal computer

Begin

millimeter

Operating System Kernel Tail A single thread of control in a process with multiple branches of control contained within a single process space

Exit I/O IPC

Multiple threads of control within a single process space. Additional threads were launched under the control of the program instead of the operating system

Figure 3.17 Basic operating system functions: memory management (MM), process control (PC), interprocess communication (IPC), and input/output control (IO)

System hardware and computer architectures are discussed in more detail in Hennessy et al. (2006).

3.4.4 Process control The operating system kernel is a process that makes privileged use of ICT resources. It has full access rights to all physical resources, has its own protected address space for its data memory, and runs the CPU in a special mode called supervisor mode. The kernel controls the access rights to physical resources for all other processes; controls the process's access to memory and controls the process's access to input and output devices. It creates an execution environment for processes to run, and sets up an address space for each application process outside of kernel space to run safely, and protects processes from mutual interference. Each process starts with a single thread of control, a single path through multiple branches and loops of control in a process. Current control threads can be split into multiple control threads, so they can have more control threads than the CPU, even within a single process space. This allows one thread to be active while others can be temporarily prevented from accessing a resource. The benefit of creating multiple threads within a single process, rather than multiple single-threaded processes, is that creating a second thread within the same process space is more efficient because it does not require creating an entirely new execution space. However, the disadvantage is that the control of threads within a process space and the communication between threads is under the control of the application process, not the kernel. That is less robust. The kernel can terminate threads of control and free their resources, such as memory and file links. Schedules process control threads to run on the CPU. The kernel maintains the state of the process, the list of processes waiting to be executed, a list of open file descriptors, and the status of each process, such as: B. active, terminated, and stuck. It also coordinates multiple processes and manages communication between processes. There are often more executable (or executable) processes, that is, processes waiting and ready to use the CPU, than the number of CPUs available to run them (Figure 3.18). Running processes can block waiting for resources. Sometimes processes do not complete yet, but a higher priority process is added. Some kind of multitasking planning is required. With static scheduling, all scheduling decisions are made before execution and all runnable processes are treated equally. With a round robin, also called a preemptive task scheduler, each process is given a fixed amount of time to use the CPU, too.

Smart devices and services

110

New Task Priority = Hello Task Size = Hello

List

new homework

List

Priority = Lo Task Size = Lo

The number of bars shows the number of CPU cycles required

Wait

Planes Planes

Wait

Run

final task

Figure 3.18 Scheduling multiple tasks that exceed the number of available CPUs

short for some processes, too long for others. Processes that terminate shortly can waste CPU cycles and interrupt40 a running process by moving it from a running context to a waiting context. Context switching41 wastes time and resources. In contrast, in a non-preemptive process scheduler, processes can run to completion, there are fewer context switches, no incomplete tasks that require a context switch. However, running processes can significantly delay other processes that are waiting to run. Dynamic programming uses decisions at run time. For example, in priority scheduling, runnable processes with the highest priority are executed first. To prevent a high-priority task from running indefinitely, a scheduler lowers the task's priority over time.

3.4.5 Memory Management The second key function of an operating system is memory management. Processes are allocated to virtual memory or an address space is allocated to primary physical main memory (RAM). The operating system kernel defines a separate area of the address space for each process. A process associated with an addressable space cannot directly access another space. The areas do not overlap and are separated from each other by buffer zones that are not addressable by the process. A region can have different parts, such as B. Read/Write (RW) access, e.g. for example, heap or stack, or read-only (RO), for example. B. Program Code Regions (ROs). Primary storage used by processes can be backed up or reallocated to more permanent secondary storage such as ROM or disk: this involves the actual copying of data from primary to secondary. Hardware, such as hard drives, network device interfaces, or graphics card interfaces, appears in a process as an array of data bytes, i. h as secondary memory data associated with the primary memory address space. This secondary memory allows programs and data to access more memory than is available in main memory; this is particularly interesting for low-resource devices. A process generates a page fault when it attempts to access data that is not available in main storage. Demand paging is commonly used to recover from a page fault, i. h to access data that is not in main storage. Requires data

40 interrupts also take time away from executing processes on the CPU. Interrupt latency is the time between when an interrupt is generated and when the operating system services the interrupt. 41 (Process) Context switch refers to a CPU switch between the state of an active process and that of another waiting process. To do this, a CPU must save the state of the active process, remove the state information from memory, retrieve the state of another process in memory, and start it.

Virtual machines and operating systems

111

copied from secondary storage to main storage. If there is no free space on the primary storage, an existing area of primary storage must be changed to secondary storage.

3.4.6 Input and output I/O devices can be referenced directly in the program as addresses or pointers to memory in the kernel mapped to the memory of the device interface card. This data can be read and written by special calls (from the operating system). File handles in the C programming language or I/O streams in C++ or Java are used to provide a high-level program API to encapsulate lower-level system I/O calls. However, this is inflexible, the program would have to be modified every time another device is introduced to get the output or input, e.g. B. to send to a printer instead of a video device. I/O support for all devices must be supported by all programs. A more effective design is to define generic I/O devices (also logical or virtual) that the operating system can remap to a specific I/O device.42 To define a remote I/O device, a standard stream, configured with a port number associated with a remote computer and a process is created. The network appears to a process as another I/O channel connected to a network interface called a socket.

EXERCISES 1. Discuss the difference between abstraction and virtualization based system designs. 2. Compare and contrast the following service access architecture models: client-server model, application server model, middleware service model, service-oriented computing model, and peer-to-peer model. 3. Discuss whether the message broadcast or flooding techniques used in P2P networks to locate unknown P2P nodes can also be used more generally for UbiCom service discovery. 4. What is a multi-tier server model? Compare a thin client server model with a two-tier thick client server model. Give some examples of models of three, four, five and six levels. 5. Describe three different layout and partition designs: (a) communication; (b) processing; and (c) data resources. 6. Describe the benefits of a proxy-based service access model; Draw a proxy-based layout to access mobile services. 7. Compare and contrast the following system component interaction paradigms for UbiCom system components: procedure call (remote), object-oriented interaction, layered network interaction, pipes and filters, shared repository. 8. Characterize a P2P model The P2P model requires a more complex synchronization compared to the client-server model, why? 9. Inform about the benefits of system designs that separate coordination or control from computing. Then discuss the advantages and disadvantages of object-oriented vs. event-driven vs. Blackboard repository coordination. (More exercises are available on the book's website.)

For example, in the Java Core API, three standard I/O devices are defined as static system object methods: System.in, System.out, and System.err.

112

Smart devices and services

References Allard , J. , Chinta , V. , Gundala , S. and Richard G.G. (2003) Jini meets UPnP: an architecture for Jini/UPnP interoperability. In Proceedings of the Symposium on Applications and Internet (SAINT'03), pp. 268-2 Alonso, G., Casati, F., Kuno, H. and Machiraju V. (2004) Web services: concepts, architectures and applications. Berlin: Springer Publishing House. Anderson , DP , Cobb , J , Korpela , E , Lebofsky , M and Werthimer , D (2002)[Email protected]: an experiment in computing public resources. ACM Notices, 45(11): 56-61. Androutsellis-Theotokis, S. & Spinellis, D. (2004) A review of peer-to-peer content distribution technologies. ACM Computer Surveys, 36(4): 335-371. Bernstein, PA (1996) Middleware: a model for distributed system services. ACM Notices, 39(2): 86-98. Birrell, A. D. and Nelson, B.J. (1984) Implementation of remote procedure calls. ACM Transactions on Computer Systems, 2(1):39-59. Bucchiarone, A. and Gnesi, S. (2005) A survey of languages and models for service composition. In International Workshop on Web Services Modeling and Testing (WS-MaTe 2006), pp. 51–63. Carman, M., Serafini, L., & Traverso, P. (2003) Web Services Composition as Scheduling. In International Conference on Automated Planning and Scheduling, ICAPS'03, Web Services Planning Workshop, pp. 1636-1642. Chakraborty D, Joshi A, Yeshaand Y, and Finin T (2006) Towards the discovery of distributed services in pervasive computing environments. IEEE Transactions of Mobile Computing, 5(2):97-112. Chan, ATS and Wan, D.K.T. (2005) Mobility web services in your pocket. In Proceedings of the IEEE International Conference on Web Services (ICWS'05), pp. 159-166. Coulouris, G., Dollimore, J. & Kindberg, T. (1994) Distributed systems, concepts and designs. 2nd Edition Reading, MA: Addison-Wesley. Dan A, Davis D, Kearney R et al. (2004) On-Demand Web Services: Automated Management Powered by WSLA. IBM Systems Magazine, 43(1):136-158. DiBona, C., Ockman, S. and Stone M. (eds.) (1999) Open Sources: Voices from the Open Source Revolution. New York: O'Reilly. Dickman, L. I. (1973) Small virtual machines: a survey. In Proceedings of Workshop on Virtual Computer Systems, Cambridge, MA, pp. 191-202. Emmerich, W. (2000) Software and middleware development: a roadmap. In Proceedings of the Conference on the Future of Software Engineering, pp. 117-129. Eugster, P.Th., Felber, P.A., Guerraoui, R., and Kermarrec, A-M. (2003): The many faces of publishing/subscribing. ACM Computer Surveys, 35(2): 114-131. Fielding, R. T. (2000) Architectural styles and design of network-based software architectures. PhD thesis, University of California, Irvine. Foster, I. and Iamnitchi, A. (2003) On death, taxes, and the convergence of peer-to-peer and network computing. In F. Kaashoek and I. Stoica (eds.) Proceedings of the 2nd International Workshop on Peer-to-Peer Systems (IPTPS'03). Berlin: Springer Verlag. Foster, I., Kesselman C., & Tuecke, S. (2001) The anatomy of the network: enabling scalable virtual organizations. International Journal of High Performance Computing Applications, 15(3): 200-222. Garlan, D. and Shaw, M. (1993) An introduction to software architecture. In V. Ambriola and G. Tortora (eds.) Advances in Software Engineering and Knowledge Engineering, vol. 2. New York: World Scientific Publishing Company, pp. 1-39. Learned, D. and Carriero, N. (1992) Coordination languages and their importance. ACM Notices, 32(2): 97-107. Goodman, D. G. (2000) Wireless Internet: Promises and Challenges. IEEE Computers, 33(7):36-41. Hennessy, JL, Patterson, DA & Arpaci-Dusseau, AC (2006) Computer Architecture: A Quantitative Approach. 4th Edition New York: Morgan Kaufmann. Hingne V, Joshi A, Finin T, Kargupta H, and Houstis E (2003) Towards a ubiquitous grid. In Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS'03), pp. 207-335. Hord, R. M. (1999) Understanding parallel supercomputing. New York: IEEE Press.

references

113

Johanson, B. and Fox, A. (2002) The event cluster: a coordination infrastructure for interactive workspaces. In Proceedings of the 4th IEEE Workshop on Mobile Computing Systems and Applications, pp. 83-93. Keen M, Acharya A, Bishop S, et al. (2004) Patterns: implementing an SOA using an Enterprise Service Bus, IBM Redbook, retrieved from http://www.redbooks.ibm.com/redpieces/pdfs/sg246346.pdf, accessed October 2007. Kindberg, T. and Fox , A. (2002) Systems Software for Ubiquitous Computing. IEEE Generalized Computing, 1(1):70-81. Krauter, K., Buyya, R. & Maheswaran, M. (2002) A taxonomy and overview of network resource management systems for distributed computing. Software Practice Experience, 32:135–164. Kumar, R., Farkas, K..I., Jouppi, N.P. et al. (2003) Heterogeneous Single-Core ISA Architectures: The Potential for Reduced Processor Performance. In Proceedings of the 36th Annual IEEE/ACM International Symposium on Microarchitecture, Washington, DC, pp. 81–92. Menasce´, D.A. (2005) MOM vs. RPC: Communication models for distributed applications. Internet Computing IEEE, 9(2):90-93. Meyer, B. (1998) Object-oriented software construction. Englewood Cliffs, NJ: Prentice Hall. Milojicic, D.S., Kalogeraki, V., Lukose, R., et al. (2002) Peer Computing. HP Lab Technical Report HPL-2002-57. Available at http://www.hpl.hp.com/techreports/, accessed November 2007. Montresor, A. (2004) A robust protocol for building pairwise overlap topologies. In Proceedings of the 4th International Conference on Peer-to-Peer Computing, pp. 202-209. Papazoglou, M.P., Traverso, P., Dustdar, S., et al. (2007) Service-oriented computing: state of the art and research challenges. IEEE Computers, 40(11):38-45. Popek, G. J. and Goldberg, R. P. (1974) Formal requirements for third-generation virtualizable architectures. ACM Notices, 17(7): 412-421. Pressman, RS (1997) Software Engineering: A Practitioner's Approach. Maidenhead 4th Edition: McGraw-Hill. Satyanarayanan, M. (2001) Pervasive Computing: Vision and Challenges. IEEE Personal Communications, 8:10-17. Smith, J. E. and Nair, R. (2005) The architecture of virtual machines. Computer Science, 38(5):32-38. Van der Aalst, W.M.P., Dumas, M. and ter Hofstede, A.H.M. (2003) Composition languages for web services: old wine in new bottles? In Proceedings of the 29th Euromicro Conference, pp. 298-305. Wyckoff, P., McLaughry, S.W., Lehman, T.J. and Ford, D.A. (1998) T.Spaces. IBM Systems Magazine 37(3): 454-474.

4 Mobile smart phones, cards and device networks 4.1 Introduction Smart user devices in this chapter focus on token or card sized devices and pads where the locus of control for user interaction is with customizable systems. Mobility is an important feature of these smart devices.

4.1.1 Chapter Overview Table 3.1 provides an overview of the various mobility concepts. This section continues the discussion of intelligent mobile services as an extension of the distributed service models given in Section 4.2. Next (Section 4.3) we discuss operating system support for mobile computing and communication devices, including resource-constrained ICT resource management and power management. Then smart card type devices are discussed, which have much more limited resources and specific applications than smartphone type devices. Finally, some types of device networks are covered. Some issues related to mobility are covered in other chapters, e.g. B. wearable devices (Section 5.4.3), implanted devices (Section 5.4.4), automotive devices, i. h Robots (Section 6.7), Smart Dust (Section 6.4), Location - Mobile Awareness (Section 7.4) and Mobile Communications (Section 11.6.5).

4.2 Mobile smart devices, users, resources and code Users are inherently mobile, e.g. B. Users can move between Internet exchange points to log in and access web-based content and email anywhere, anytime. Users can carry personalized connected mobile devices to access services filtered based on their personal preferences and to learn and adapt to their location context (Chapter 7). Other types of inanimate hosts, such as transport vehicles, can also function as mobile hosts. each of the most important

Stefan Poslad

116

Smart phones, cards and device networks

The components of a UbiComm system (Sections 3.1.2, Section 3.2) can be mobile:1 virtual processing and services at the operating system and application level, code, access devices,2 hardware, and data resources.

4.2.1 Mobile service design Mobile service design is based on the basic design of smart devices and services, but is a more specialized variant of it. To simplify application and user access, mobility, e.g. B. the location and routing of mobile users and the routing of data to mobile recipients are made more transparent to applications and users. There are three types of visibility for mobile services: (1) Virtual User Environments (VUEs); (2) Mobile Virtual Terminals (MVT); and (3) virtual resource management (VRM). UVE gives users a consistent view of their work environment, regardless of location and device type, e.g. B. MobiDesk (Baratto et al., 2004) and Virtual Home Environments or VHE (Moura et al., 2002). MVT preserves the running state of the terminal for recovery to new locations, including running processes and subscribed services, e.g. B. mobile agents (Section 4.2). VRM allows mobile users and endpoints to maintain access to resources and services by automatically requalifying links and offloading specific resources or services to enable load balancing and replication (Bellavista et al., 2001). Mobility within a homogeneous network or a network of networks can be supported at the Internet or network layer of the network protocol stack, so that mobility is transparent to applications (Section 4.2). Mobile devices can use a variety of heterogeneous wireless communication protocols.3 Mobility transparency is a major challenge in heterogeneous networks. Application processes could perform a soft handover from one type of network to another with a finite delay, but this is easier if supported by the operating system. The transition from networked use to the standalone device should be smooth. Wireless connectivity is still spotty, with different protocols around the world, blackouts while on the move, and incomplete coverage, especially in remote areas, in some buildings, or in the air. Relying on an always-on cellular connection can be very frustrating for some data applications, assuming such a connection. In short, cellular telephony processes must support an ephemeral connectivity model (Section 3.3.3.9). Mobile devices can create new local data that may be sensitive to business or personal. A denial of service (DoS) can occur when a mobile device is stolen or left behind. There are two possible solutions to deal with DoS. First of all, you can use backups, occasional wireless synchronization or data cable synchronization. Second, a remote access model can be used to support a UVE-like distributed virtual desktop on the mobile device, so that data that appears to be local is managed remotely, e.g. B. MobiDesk (Baratto et al., 2004). MobiDesk adds a virtualization layer that supports a stateless, thin client model, but requires an always-on connection on demand. Data synchronization is required for both techniques. In applications, services, devices, computers, it must be clear to the user what is being synchronized. With the synchronization in the foreground, the user initiates the synchronization and selects the data that he wants to synchronize. With background synchronization, a specific event triggers scripted data selection and subsequent data synchronization. In

Other than the physical environment, which is usually not mobile unless it is a natural disaster. Although access devices, computing resources, and peripherals are primarily mobile, middleware devices such as routers can also be mobile and positioned to optimize coverage, etc. eg B. OSGI model (Section 4.5.2). 2

Smart mobile devices, users, resources and code

117

Additionally, concurrency conflicts can arise when changes are made to shared data remotely and locally. Database concurrency control models can be used, but this is challenging as data is distributed and data storage and media can be heterogeneous unless a VUE mobility model is used. Eventually, when mobile devices become unreachable, they could be set to wake up and wipe all important data once they connect to an unknown network.

4.2.1.1 SMS and mobile web services One of the oldest but most widespread data applications for many mobile phone users is the short message service (SMS). SMS is a message transport service that supports a reliable two-way connectionless messaging protocol. It is a very simple protocol that supports sending and receiving and is often used for notifications. Its main limitations are text orientation, text size is limited to 160 characters or less, message latency is one minute or more, application-level security is poor, and it is difficult to link messages to interactions and transactions. .4 Due to the limitations of SMS, a more flexible data protocol was introduced, the Wireless Application Protocol (WAP)5. The goal of WAP was to become the de facto world standard for the presentation and delivery of wireless information and telephony services: Internet on a mobile phone. WAP supports connection-oriented and interactive communication sessions and multimedia. There is no fixed message size limit, it is secure and the latency is lower than SMS. WAP supports data formatting and browsing for small screens. WAP was designed to send data wirelessly with delays, slow connections, and low-resource mobile devices (Goodman, 2000). There are also several design challenges with the Wireless Markup Language (WML) that WAP uses, as well as having such significant differences from HTML that mobile developers almost have to learn a different markup language. WML uses a user interface metaphor of a deck of cards. The compressed WML platform must not be larger than 1.4 KB. WAP devices are often not 100% compatible with WML or support 100% of its features. The WML rendering on some microbrowsers makes navigation difficult. Sometimes the functions of the device cannot be set. Cache issues are also a problem. WAP also faces competition from a mobile information service called i-mode, launched by Japan's NTT DoCoMo in February 1999 and based on a proprietary technology, the DoCoMo packet communication network protocol (Enoki, 2001). The main advantages over WAP were that i-mode used a content language, compact HTML (cHTML), which was much more similar to HTML than WML. Web-based services were cheaper to use because they were packet-based rather than time-based. I-mode was also developed for use with higher resolution mobile devices. Mobile web services design often uses a three-tier architecture of thin client, client-proxy, and server (Section 3.2). The thin client architecture simplifies the functionality of mobile devices: they are used only to access information and can run a web mini-browser or microbrowser (Lawton, 2001), but application functionality is limited (see Figure 4.1). A client proxy (Section 3.2.2.4) is used to offload heterogeneity handling of content adaptation to heterogeneous devices, microbrowsers, and content languages. With a passive approach to content personalization, there is no automatic adaptation to the capabilities of the access terminal and web browser, users must choose the content format. To adapt content to a variety of devices, the following content properties must be defined: the I/O capabilities of the access device, the supported content language types; the type of presentation, e.g. B. Microbrowser used to render the content, as well as the design and style of the content presented.

Through gateways it is possible to exchange messages with other systems such as email, web, etc. WAP Open Forum was founded in 1997 by Ericsson, Motorola, Nokia, etc. In 2002, OMA, the Open Mobile Alliance, was founded by some 200 companies. In 2003, the WAP Forum and other similar forums were integrated into OMA, http://www.openmobilealliance.org/, accessed July 2007.5

Smart phones, cards and device networks

118

Microbrowser controls used to drag content to the terminal

Access nodes for wireless stations

service provider

Hotel Application Gateway Attractions

HTTP

landline

Access nodes for wireless stations

eat/drink out

Determining the location of tours/maps, e.g. B. GPS Figure 4.1 Example of a thin client server architecture, using a microbrowser running on a mobile device to retrieve content over a wireless network

The content adaptation can be static, prepared for the most common content feature sets for a lowest common denominator approach, or dynamically adapted to each content feature set. Information about the form factor capabilities of different devices, the number of x-y pixels, and color depth is needed to scale and adapt content to the limitations of a specific screen. This is especially true when considering multi-publishing, author-publish-once-many applications. The Device Independence Group (W3C) created the Composite Preference/Capability Profile (CC/PP) specification6 in the RDF/XML language. In collaboration with the W3C, OMA has created a mobile device vocabulary for CC/PP called User Agent Profile (UAProf).7 As mentioned above, there are different content languages for use in microbrowsers, e.g. B. WML and cHTML. The development of the W3C XHTML/CSS Recommendations for the mobile web suggested that content languages would be standardized (Pashtan, 2005). In fact, the situation was made worse because XHTML/CSS had even more combinations than WML8 and because experience has shown that it is extremely difficult to develop an XHTML portal for devices without detailed content customization solutions to dynamically fix a variety of browser bugs. one's

See http://www.w3.org/Mobile/CCPP/ proposed by the W3C Ubiquitous Web Applications Working Group, accessed January 2007. Note that other W3C groups have developed terminal definitions, e.g. B. the Mobile Web Initiative Working Group, http://www.w3.org/2005/MWI/DDWG/, which proposed the Device Description Repository (DDR). CC/PP and DDR are similar. 7 Unfortunately, adoption of CC/PP and UA profiles by device manufacturers has been relatively low to date. Profiles can be hard to find, often invalid, or just plain wrong. 8 To date there are four versions of XHTML (XHTML 1.0, XHTML 1.1, XHTML-Basic, XHTML 2.0) and OMA has been working on a mobile profile called XHTML-MP. Version compatibility is problematic and is further accentuated in the mobile space where XHTML-MP is no longer compatible with the versions developed by the W3C.

Smart mobile devices, users, resources and code

119

From an operator's perspective, wide permutations of browser implementations and content languages are a key constraint impeding the growth of the mobile Internet or increasing its complexity. Content layout defines whether an entire article, e.g. B. a full story is displayed on a desktop computer, on a mobile phone or just its title; whether the menu should be placed on the right for the desktop screen in landscape mode and on the top for the vertical screen, what kind of navigation should it be and how to access multiple screen contents. Because mobile devices have limited local storage, a local content management strategy is required. Content may expire automatically or must be manually removed.

4.2.1.2 Java VM and J2ME J2ME, the mobile Java9 2 environment, supports the development of large client-server systems and stand-alone system designs on which code is developed. The mobile device can be networked with servers implemented with Java Standard Edition (J2SE) or Java Enterprise Edition (J2EE). The J2ME platform arose from the need to define a computing platform capable of hosting consumer electronics and embedded devices. J2EE, J2SE, and J2ME applications are developed to run in a Java virtual machine (see Figure 4.2) to support code portability between heterogeneous devices. Thus, as long as an appropriate subset of the Java API is used, the same code can be written once and read and executed on many different devices. Rather than being structured as a single, monolithic platform with a large footprint for devices, J2ME is a multi-tiered organization of software packages of three types: configurations, profiles, and optional packages. A configuration such as CDC or CLDC provides basic services for

Server Optional packages for servers

Java 2 Enterprise Edition (J2EE) Option Packages

Java 2 Micro-Edition (J2ME)

desktop machines desktop machines

High-end high-end consumer device devices

Low-end consumer consumer devices

Basic Profile Java 2 Standard Edition (J2SE)

Intelligent SmartCards

Personal profile Foundation profile

MIDP

Centers for Disease Control and Prevention

CLDC

Java Card-API

KVM

card virtual machine

Java virtual machine

Figure 4.2: J2ME uses a VM to support a variety of devices

In the early 1990s, the Green Project at Sun developed the Oak programming language, later renamed Java, for programming consumer electronic devices, but it was used more for developing desktop and server-based applications. In 2000, the J2ME Platform Specification (JSR 68) was released, going back to the roots of Java to support low-resource programmable devices.

120

Smart phones, cards and device networks

a broad class of devices and virtual machines. A profile such as the CLDC Mobile Information Device Profile MIDP supports higher level services for more specific classes of devices. An optional package adds specialized services that are useful on many types of devices but not required on all, e.g. for example, those defined in other J2ME extension specifications JSR.10 This organization promotes both reuse and efficiency by allowing developers to compose a software stack that meets both target device capabilities and resource requirements of the applications. The J2ME platform divides 11 devices into two distinct categories or configurations (see Figure 4.2). First, the Connected Device Configuration (CDC) devices are defined. These are devices with more than 2MB of RAM and ROM that support always-on networks that have special purpose or limited functionality. They are not general purpose calculators, e.g. B. Set-top boxes, Internet TVs, Internet-enabled display phones, high-end communicators, and in-car navigation/entertainment systems. These devices generally have higher resource user interface facilities and more processing power. Second, the Connected Device Limited Configuration (CLDC) devices are defined. These are devices with 160-512 KB memory and support intermittently connected mobile information devices with limited GUI, e.g. B. cell phones, two-way pagers, PDAs, and organizers. CLDC also supports card-based ICT devices with limited ICT resources and no GUI.

4.2.1.3 .NET CF Windows Mobile12 is a variation of Microsoft's Windows operating system for minimalist computers and embedded systems. Windows Mobile is now based on a significantly different kernel operating system, instead of simply being a "scaled down" version of desktop Windows. Windows CE is a modular13 operating system that supports several classes of devices. Some of these modules provide subsets of the functionality of other components, e.g. different levels of windowing support (DCOM versus COM), others that are mutually exclusive, and others that add additional special features. The .NET Framework is an integral component of Microsoft Windows for building and running WS and software applications (see Figure 4.3). The .NET Framework consists of two main parts: the common language runtime (CLR) and a unified set of class libraries, such as ASP.NET for web applications and services, Windows Forms for intelligent client applications, and ADO.NET for loosely coupled data. . access. Code written on the .NET Framework is known as managed code. This refers to the Common Language Runtime (CLR), which provides assertion code that handles certain common errors that affect Win32 programmers. Managed code must not have malformed pointers, cannot leak memory, and supports strong code type safety. Managed code is compiled to a combination of MSIL (Microsoft Intermediate Language) and metadata.

10 There are additional J2ME specifications, such as JSR-135 Mobile Media API, JSR-172 Web Services Specification, and JSR-82 Bluetooth API. Johnson (2006) reports experience using JSR-82, while many phones support Bluetooth, few devices specifically support J2ME and JSR-82. .11 This distinction is a bit blurred as technology continues to make it possible to pack more and more power into smaller and smaller devices. Windows Mobile 12 now provides a unified underlying platform for three types of devices: Microsoft smartphones, Pocket PCs, and Pocket PC phones. There are several other related names. Windows CE, Windows Mobile, and Pocket PC are often used interchangeably. This practice is not entirely correct. 13 WinCE or Windows CE is also a real-time operating system with deterministic interrupt latency. It supports 256 priority levels and is thread-based to support concurrent scheduling.

Smart mobile devices, users, resources and code

VB.Net

121

Handling other C/C++ MSIL languages

Byte code

CLR runtime (VM) services, e.g. B. Security Windows operating system

Other languages

Java

JRE Runtime Services (JVM), e.g. B Mac Security

To win

Unix

linux

Figure 4.3 .NET virtual machine versus JRE virtual machine

These are merged into a pre-execution environment (PE) file, which can then be run on any CLR-enabled virtual machine.14 When executed, the JIT begins compiling the IL to native code and executes it. The Microsoft .NET Compact Framework (CF) is a rich subset of .NET designed specifically for resource-constrained devices such as PDAs and smartphones. Its class libraries are a subset of .NET (about 25%). .NET CF also has some additional mobile-specific libraries: IrDA support, SQL Server CE, and device-specific controls. .NET CF runs on a powerful JIT compiler for mobile devices. A thick client server design requires the .NET Framework to be installed on a mobile device.

4.2.2 Mobile code The code is normally designed to be downloaded from a remote service point. Installation requires code installation configuration on each platform. This is automated by a makefile, e.g. for example, the installers can tell the makefile on the file system to install the code. Once installed, it often remains at one point in the ICT infrastructure, but can be maintained and updated. Service access devices can also download new operational capabilities at runtime without the need to store any service support that may be required in advance; this reduces the need for resource-rich service access devices in dynamic environments. This paradigm allows vendors to keep code on consumer devices with a network connection, e.g. B. to update and repair without the provider having to ship physical media to customers. Unlike mobile computing, where devices or users move, and unlike static code, where code resides in one place, mobile code changes the network node where the program runs (Fuggetta et al., 1998). Code mobility is an important requirement for system extensibility to support operation in open dynamic environments, especially when using access devices with limited resources. Commonly used mobile code representations include Java, PostScript, which instructs printers to create images, and many other declarative languages that contain instructions that can be interpreted at a remote service access point. While the first mobiles

14 The .NET CLR and JVM approaches are similar in that they both use a VM that runs machine-independent code. JVM is designed to be platform independent (for multiple operating systems) and primarily uses a single programming language (Java). A separate JVM is required for each operating system and device type. CLR is designed to be language independent (development supports multiple languages like C++, VB, C#, etc.) but for a single underlying operating system, even though there are multiple versions of Windows operating systems.

122

Smart phones, cards and device networks

Devices that need to be activated and access code updates while connected to a PC through a peripheral connection, many of today's mobile devices can be activated and updated via code over the air (OTA) or over WWAN. Brooks (2004) provides an overview of mobile code paradigms based on where the code runs and who determines when the mobility occurs. These include: client-server interaction and remote testing, where client-side code must be downloaded (implicitly) before it interacts with server-side code; code on demand, where clients (explicitly) download and run code as needed; Process migration, in which processes are moved from one node to another to balance the processing load; mobile agents, in which the program code and its state move from one place to another according to their own internal logic; active networks, where packets moving across the network reprogram the network infrastructure. The benefits of mobile code include increased system flexibility, scalability, and reliability, but this is mitigated by the increased security risks associated with potentially malicious or buggy code downloaded to devices and frequent disruptions to the ICT activities of the consumer, when multiple applications are updated on multiple devices. There are currently four main approaches to mobile code security (Zachary, 2003): sandboxes, which restrict the local services that the code can access; code signing, which ensures that the code comes from a trusted source; firewalls that limit the computers that can access the Internet; and Proof-Carrying Code (PCC), which contains explicit proof of its security.

4.2.3 Mobile devices and mobile users The mobility of devices can be viewed from several dimensions: (1) in terms of physical dimensions); (2) whether or not the device is mobile, or whether the type of host to which it is connected is mobile; (3) with respect to what types of host mobile devices may be linked; (4) how devices connect to a host; and (5) in terms of when mobility occurs. Each of these will be discussed in turn. Mobility is somewhat dependent on physical dimensions, the smaller a device is, the less energy is required to move it, increasing the degree of mobility. Several different shapes for the UbiCom devices discussed in sections 1.4.1.1 and 1.4.1.2 that could be mobile include dust, tabs, pads, and skins. On the one hand, we have the meter-sized UbiComp devices, which are often centimeters thick and too heavy for one person to carry. However, if the thickness could be reduced, thus making the material more flexible, then it would behave like a skin and could be made more wearable; its size may be reduced by rolling or folding during transport. Devices such as screens and organic cloth (Section 5.3.4.3) can act as mobile masks. The second largest mobile device is the decimeter-sized pad devices such as a laptop, optical scanner, printer, etc. Some of them run MTOS, some of them run ASOS. The second smallest mobile devices are one-inch-sized pad devices, such as cell phones and smart cards. Mobile phone type tablet and tablet devices tend to be expensive ICT devices with higher resources and typically run MTOS (Section 4.3). Card-type Tab devices are typically much smaller resource ICT devices and are controlled by a more specialized CardOS (Section 4.4.1). The smallest type of mobile device is a dust-sized device (Section 6.4). Devices may be inherently mobile and use their own rechargeable or renewable energy sources to power them, e.g. B. a robot (Section 6.7). Devices can be mobile because they are connected to a host that can be mobile. A mobile host can be animate or inanimate. Devices can be mobile because they are linked in some way to an embedded mobile host, such as a living thing or an artifact of the physical world, such as a vehicle, or they can be attached to non-living things that move, for example, tiny things or microdust. . large devices can bind to air and liquids flowing through the physical environment driven by physical forces.

Operating systems for mobile computers and communication devices.

123

There are three basic ways in which mobile devices can be physically linked to mobile hosts: escorted, surface-mounted, or integrated into a host's structure, e.g. B. an embedded controller built into a host device. Accompanied refers to an item that is loose and accompanies a mobile host, e.g. B. a mobile phone that can be carried in a pocket or pocket but is easily misplaced. A device can be surface mounted on a mobile host. If the mobile host is a human, we call the carrier surface carrier. Smart Paint can be sprayed on a surface (Section 6.4.4). Embedded means that a device can be permanently integrated into a mobile host, e.g. B. a brake control system integrated into a vehicle. When a device is attached to a human or animal in this way, we call it an implant. The different stages of the device's operational life cycle where mobility can occur are as follows. A device can be moved from the manufacturing site to a permanent installation site; it can be mobile between sessions but static during sessions; can be mobile (roaming from home) during user sessions but connected to a home location; you can roam free (without limits) without a home.

4.3 Mobile computer and communication device operating systems Desktop computers and ICT-intensive mobile devices such as smartphones (thick client devices) often require more complex multi-process control and the use of a multitasking operating system (MTOS) to support execution. of multiple users and applications.

4.3.1 Microkernel Designs In a macrokernel or monolithic kernel operating system (Section 3.4), all system utilities such as hardware-related drivers, memory management, process support, process scheduling, the stack of network protocols and the file system reside on a single, large system. , kernel system (Liedtke, 1996). The main advantage of the monolithic kernel system is that it is more efficient for a uniprocessor system since fewer context switches are required. Context switches are only required between processes and kernel utilities, and not between processes in general. The main disadvantage of a monolithic kernel system for systems with few resources, such as mobile systems, is that the kernel usage is quite extensive and requires a lot of system resources. Also, the monolithic kernel is potentially more complex and has more points of failure and may require more updates, e.g. B. to support the addition of new hardware. Rather, only the basic parts of the operating system, such as basic memory management, (limited) process management, and inter-process control, are supported by a microkernel. The potential advantages of a small kernel are that it is easier to maintain and more robust in a low-resource environment. It can still work even if non-kernel system utilities fail. Disadvantages are that there can be more context switching between the application, non-kernel utility, and kernel utility process execution contexts, which can reduce performance when running on uniprocessor systems.15

4.3.2 Mobility Support Some operating systems originally designed for resource-poor devices, such as mobile devices, relied on building a stripped-down version of PC-style MTOS, eg. B. Windows Mobile, while other designs for an operating system were specifically aimed at a resource-efficient mobile device and other special features of the

15 The macro vs. microkernel debate was discussed in DiBona et al. (1999) with Linus Torvalds, creator of Linux, advocating a macrokernel design, and Andrew Tanenbaum, creator of Minix, advocating a microkernel design.

124

Smart phones, cards and device networks

land. Mobile device design continues to evolve. Some aspects of mobile operating system support are even suggested to produce a cost effective, resource and energy efficient PC design. In addition to normal operating system ICT support, mobile devices have several more specialized requirements, including HCI for small mobile user interfaces (Section 5.2.3), heterogeneous communication support (Section 11.7.5), intermittent connections (Section 3.3.3.9), data management (Section 12.2.9), mobility support (section 11.6.5) and power management (section 4.3.4). The operating system kernel should be small for a low-resource device. Basic support for memory management is required to prevent memory leaks and free up system resources as soon as they are no longer needed. Good resource reuse strategies are crucial as resources are limited. There are several options for multitasking support. Multitasking is useful to support real-time communication performance to talk simultaneously, set alarms, and run and access data and applications on the phone, e.g. B. Symbian OS.16 Alternative to The system can schedule one task at a time, wait for it to finish, and then switch to another, i. h Non-preventive task scheduling, e.g. B. Palm OS. Small mobile devices typically do not use magnetic disk-based secondary storage for persistence due to relatively slow access speeds, damaged moving parts, and higher power consumption. Persistent memory comes in the form of Flash ROM to store files and data, however Flash memory is slower than RAM. Mobile devices tend to boot from ROM. However, flash memory consumes less power, so the battery life of the devices can be longer. Note that in the past, certain types of ROMs had a shorter lifespan in terms of the number of reads and writes they supported before failing.

4.3.3 Devices with limited resources Operating systems for devices with limited resources, e.g. B. Mobile handheld devices have to deal with low primary storage, low secondary storage capacity, slow CPU, limited input/output, and operation in a low-power mode17 due to a finite power supply. Strategies to deal with low memory include data compression, offloading a device's data storage (thin client) to remote network servers, simply overwriting older data, and using higher capacity secondary storage. Strategies for adapting computing to a slow CPU include offloading computing from local servers to remote networked servers, using variable voltage CPUs, and using power-based and predictive process OS scheduling (Section 3.4). Strategies to reduce the power required for communication include the use of communication strategies in which higher-power servers and other devices initiate transmissions18 and the use of short-range transmissions. Satyanarayanan (2001) proposes different strategies to adapt to resource constraints. First, the system can automatically guide applications to change their behavior and use less of a scarce resource. This change typically decreases the user-perceived quality or fidelity of an app. Second, a client can ask the environment to guarantee a certain level of a resource. This is the approach typically used by reservation-based QoS (Section 11.7.1). From the customer's perspective, this effectively increases the supply of a scarce resource to meet customer demand. Third, a client may suggest a corrective action to the human user of the application and it is up to the user to follow it or not. Personalization of the UbiCom system in general is considered in more detail in the context of context perception (Section 7.2.4).

sixteen

Symbian Developer Network at http://www.symbian.com/, accessed August 2007. For example, a high intensity light bulb typically consumes 100 W/h, a CFL 10 W/h, a laptop 10 W /h, and a phone 0.1W/H. 18 The power required for transmission is sometimes greater than the power required for reception, since transmitted signals become weaker as distance increases. 17

Operating systems for mobile computers and communication devices.

125

4.3.4 Power Management Power management is essential for many embedded and mobile devices that are not connected to a permanent power source. Increases your on-site uptime before the power supply needs to be replenished and reduces the time and labor required to regularly replenish or replace nonrenewable energy sources. Power management is also an issue for devices that are permanently connected to a power source due to environmental and economic concerns. There are several strategies to reduce the power consumption of devices, including: manual overrides to allow users to specify when they no longer need to use the devices; use simple timeouts to detect long idle states and shut down devices in standby or power off mode; using more detailed techniques than on/off mode for power consumption (Section 4.3.4.1) and power conversion to store power for later use that would otherwise be wasted.

4.3.4.1 Low Power CPU If devices are fully powered on all the time, electricity may be wasted unnecessarily because, for example, the CPU, hard disk, and screen are not used. Devices could go to sleep instead. However, they must also be able to react in all situations. When powering on the device, a brief startup sequence is required. Devices should never be turned off completely, as they must set off timed alarms or handle incoming calls, allowing for many hours of operation on a single charge or set of batteries. Competing processes and users may need to be scheduled to get a fair share of battery resources (power) instead of CPU resources, for example, when power resources are running low. The CPU typically consumes 30-50% of a device's power. The power P of a CMOS processor satisfies Equations 4.1 and 4.2. P ¼ kCV 2f

ð4:1Þ

ð4:2Þ

f / ðV Vt Þ =V

where k is a constant, C is the capacitance of the circuit, f is the CPU frequency, V is the voltage, and Vt is a threshold voltage. An approximation of energy consumption is given in Equation 4.3. PðV3Þ

ð4:3Þ

Therefore, if the CPU voltage could be lowered, the power savings would be significant. Note, however, that throttling the CPU also reduces the CPU clock rate. Dynamic Voltage Scaling (DVS)19 allows the voltage of a CPU to be adjusted dynamically to save power (Pillai and Shin, 2001) (Figure 4.4). When a CPU runs at a constant speed, a minimal amount of power is consumed if the number of CPU cycles can be accurately allocated. However, jobs statistically use cycles, often completing before they have used up their allotted time. Real-time, smooth, stochastic control of the CPU clock has the potential to save even more power. CPU voltage is determined based on program response time requirements and schedules. Earliest Deadline First (EDF) scheduling allocates the CPU cycle budget per task based on profiles. Executes tasks with the earliest deadlines and a budget based on the number of cycles consumed by the task. A task is prioritized when its budget is exhausted (Section 3.4).

DVS-compatible processors include PowerNow! and Intel's SpeedStep and XScale (Pillai and Shin, 2001).

Smart phones, cards and device networks

126

application requirements

surveillance

profiler

Planning

planer

distribution of demand

Time Pressure Speed Adapter

Kernel Power Management

Speed scaling CPU profiler recognizes that work needs to be sped up

velocity

Context switching &. Save the new speed for future use

B1A1

version A1

Figure 4.4 Using Dynamic Voltage Scaling and Real-Time Soft Scheduling to Reduce CPU Utilization and Power Consumption

4.3.4.2 Application support Ellis (1999) argues that power management by the operating system alone is not optimal, applications must be involved. He proposed a wish list of capabilities for a powerful operating system and what should be exposed to the application layer. These include: decoupling the states of different devices so that they can be specified independently; Use a notification mechanism for upcoming performance related events (eg device goes into sleep mode) with the ability to react e.g. B. to store data in secondary storage. A key challenge is how to deal with the power management needs of multiple applications that may conflict with each other. In addition, the resource and performance requirements of the background tasks may differ from the foreground tasks that currently control the input and output channels of the master device.

4.4 Smart Card Devices A smart card 20 is a plastic card with embedded digital memory and (usually) a microprocessor chip, unlike cards that store data on magnetic stripes. It is reprogrammable, stores and processes data on the card, and exchanges data between cardholders and applications. Data can be stored and processed in card memory or in the microprocessor that is accessed through a card reader (Shelfer and Procaccino, 2002). Smart cards are small and easy to carry. They provide a secure data container, can be used for authentication purposes, e.g. B. as a hardware-based digital signature and can be used for payment services (Husemann, 1999).

In 1977, Motorola and Bull, the French computer company, produced the first smart card microchip. In 2006, the global chip card market was estimated at around 7 billion units. Smart cards can only be memory and not contain a microprocessor. They are used like prepaid cards to transfer cash to a vendor, e.g. B. when making a phone call at a public kiosk, and simply throwing them away when empty.

smart card devices

127

ISO contacts (used con lector)

Vcc reader contact

time

FIRST

VPP

CLK

IT'S IN

RFU

antenna memory management unit

ROM

EEPROM

Card OS contactless reader

FIRST I/O CLK

UPC

cryptographic control unit

Krypto-Motor

Figure 4.5 Contactless and contact smart cards. Contactless cards include a built-in antenna and transceiver to interface with a reader. Contact cards contain electrical contacts for a reader. A typical smart card operating system is also shown.

Many things that are in a person's wallet can potentially be replaced with a smart card, including driver's license, insurance information, chip and PIN, credit or debit bank card, travel card, and ticket , etc. Smart cards potentially represent a malicious form of privacy. -Invasive Technology or PIT (Section 12.3.4.2). Smart cards can be contact or contactless (Figure 4.5). Contact cards must be inserted into a reader and make physical electrical contact. This can be chosen for security reasons, e.g. a request for a chip (and input) PIN (Personal Identification Number). Contactless cards also contain an antenna and an RF transceiver or transponder. The card can be rotated in close proximity to a reader or base station for interaction, e.g. B. A contactless designed smart travel card reduces authorization time compared to using a contact card. Multiple types of plastic cards and applications could be combined into one multifunctional and universal smart card. Unanswered questions include whether the cardholder owns the private data, how cardholders can access their personal information, how multiple types of card data are managed, differentiated, and transmitted across multiple applications on multiple cards, and how may be the life cycle of smart. managed cards.

4.4.1 Smart card operating system The main tasks of a smart card operating system on behalf of applications are to transfer data to and from the smart card, control command execution, manage files, administer, and execute cryptographic algorithms to protect the access to stored data. and manage and run program code. The components of a typical card operating system (see Figure 4.5) are

128

Smart phones, cards and device networks

combined into a single IC chip to reduce size. The ISO/IEC 781621 standard was defined to standardize the communication protocol between the on-card application and off-card programs accessed through the reader.

4.4.2 Smart Card Development Java Card refers to the technology that enables small Java-based applications (applets) to run securely on smart cards and similar small-sized devices, such as JavaRings. A Java card is the smallest version of Java targeted at embedded devices. It is commonly used in SIM cards (used in GSM mobile phones) and ATM cards. The first Java Card was introduced in 1997. Java Card applications use the Java Card bytecode, a subset of the standard Java bytecode that runs on a Java CardVM. The J2ME base profile and configuration are still too large for a smart card, a subset of the Java bytecode used by Java KVM is used in CardVM.22 A different encoding is also used and optimized for size. This saves disk space, a necessity with resource-constrained devices like smart cards. There are techniques to overcome the memory limit on some smart cards, such as B. application code partitioning.23 Java CardVM runs on many smart cards, eg. B. the SIM card (Subscriber Identity Module) of GSM phones. SIM cards securely store the service subscriber key (IMSI) used to identify a subscriber. SIM cards allow users to switch phones simply by removing the SIM card from one cell phone and inserting it into another cell phone or broadband telephony device. Four main steps involve the development of the Java Card applet using the JavaCard development kit.24 First, the functions of the applet are specified, e.g. B. A security feature requires the user to enter a PIN; the card will be blocked after three unsuccessful attempts to enter the PIN, etc. An ID (applet ID) is requested and assigned to both the applet and the package containing the applet class. The class structure of applet programs is defined. The interface between the applet and the terminal application is defined. Java Card focuses on providing a common programming interface for developing smart card applications, however the communication between the card applet and non-card services is considered very simple, non-object oriented and rather ad hoc (Chan and Wan, 2005). ). Chan and Wan propose a WS-SOAP-based wrapper for Java Card communication between card and reader and a proxy at the reader to map WS-SOAP messages to lower-level Java Card communication messages. WSDL (and therefore UDDI) was not used on the card side to describe the services offered by Java Card, as it was considered too resource-intensive, but it can be used in a proxy-based design. Its WS architecture, called WSCard, was designed for use in an online drug ordering web service.

4.5 Device Networks The goal of a device network is to allow a variety of devices to work together. These activities include home automation such as lighting and air conditioning, personal systems, home security,

ISO/IEC 7816 is a set of standards from the International Organization for the Standardization of Smart Cards: physical properties (ISO 7816-1); contact point and dimensions (ISO 7816-2); electrical signals and low level transport (ISO 7816-3) and high level application (ISO 7816-4) Available at http://www.iso.org/iso/iso_catalogue/, accessed August 2007. 22 For example , no does not support some Java language functions (char, double, float and long types; enumerations; arrays with more than one dimension; completion; threads). 23 Similar techniques were also used in early PCs to overcome memory limitations. 24 See http://java.sun.com/products/javacard/dev_kit.html.

device networks

129

Community promotion and ubiquitous access to AV content (Section 2.3.2.1). This requires service infrastructures and home networks that are easy to install, configure and maintain; low-cost equipment and infrastructure (capital and operating costs) and useful applications. According to (Vaxevanakis et al., 2003), wireless technologies will dominate in the main home network. This is partly because of the ease of whole-home distribution without the maintenance costs of rewiring, and because the added insecurity of wireless doesn't seem to be much of a problem for users at the moment. Currently, the infrastructure for home devices is highly fragmented and far from a perfect infrastructure. First, many electronic devices are monolithic, possibly non-digital, and non-networkable. Second, heterogeneous control devices and networks dominate. Various systems have been proposed to connect and control home appliances in an integrated way, including InfraRed, X10,25 HAVi and HES (Section 4.5.1), UPnP and Jini (Section 4.5.2) and OSGi (Section 4.5.2). Smart mobile devices such as laptops and mobile phones can use WLAN, DECT and 3G mobile networks. Video transmission generally uses a different network (Section 11.6). Bluetooth, which is supported by more and more consumer devices, is another type of wireless network. Third, the inherent problems with wireless networks, such as ease of detection and limited power supply for many mobile devices, are a major concern. For example, consumers may prefer to use humans as networks26 to move storage media between devices, as this reduces energy costs for transmitting communications and simplifies the coupling of the content source and content processing application.

4.5.1 HAVi, HES and X10 Home Audio Video Interoperability or HAVi is an industry standard specification that provides standard APIs for typical home audio and video devices, e.g. B. TV, provides. HAVi also provides a discovery service to find devices connected to an IEEE 1394 network. Home Electronic System (HES) is an international standard for home automation that is being developed by experts from North America, Europe and Asia. The working group is officially known as ISO/IEC JTC1/SC25/WG1.27 HES defines the following system components. A universal interface is built into a device to communicate over a variety of home automation networks. A command language for device-to-device communications is independent of the network carrying the messages. A residential gateway, HomeGate, connects home control networks to the networks of external service providers. Standard interfaces are defined to support interoperability between application domains, such as security, lighting, power management, and for command, control, and communication in mixed-use and commercial buildings, such as residential, retail, and office buildings.

4.5.2 Device Discovery If device networks are to support open, dynamic, and heterogeneous device access and interoperability, automatic device discovery is required to simplify usage. Device detection standards and

X10 Wireless Home Networks, http://www.x10.com/ Human movement of data has been used throughout the history of personal computing. In the days of floppy drive personal computers in the 1980s and 1990s, this was known as floppy networking or sneaker networking because people wearing shoes called slippers would walk or sneak around to transfer data between different computers. Today, USB sticks are used instead. 27 http://hes-standards.org/, accessed January 2008. 26

130

Smart phones, cards and device networks

Technologies include Sun's Jini, Microsoft-led UPnP Forum, IETF SLP, DNS Service Discovery, and Bluetooth SDP (Helal, 2002). Introduced in 1998, Jini consists of three Java language protocols: Discovery, Join, and Lookup. Discovery occurs when a service searches for a browser, either using a priori information or using a multicast to which it can register. Service providers use Join to register service functions. Lookup occurs when a client or user finds and invokes a service described by its interface. The service can then be invoked by downloading it to access it locally, or by using a remote method invocation (RMI) protocol to download service access proxy software to invoke the service remotely. Jini only grants access to the Services by leasing services for a certain period of time. Before the lease expires, applicants must apply for a service lease renewal. This avoids maintenance of service connection resources for service requesters that are no longer active. Simple Service Discovery Protocol (SSDP) is a Universal Plug and Play (UPnP) protocol that uses HTTP notification advertisements to request a service type URI and unique service name (USN). SSDP is supported by some firewall devices, allowing applications to tunnel through HTTP for external access, and using SSDP makes it easy to share media between host computers and media centers. The UPnP protocol is based on HTTP/TCP or UDP/IP and XML. A control point on a UPnP network is similar to a client. It is able to recognize and control other devices. UPnP uses pure XML instead of WSDL to describe device profiles and supports high-level service descriptions in the form of a user interface that users can access via a URL. This can be loaded into a web browser and allows the user to control a device or view the status of the device, depending on the capabilities of the site. Devices can dynamically join a network, obtain an IP address, communicate their capabilities on demand, and learn about the existence and capabilities of other devices. SSDP is similar to Jini in terms of supporting three protocols for discovery, join, and search. SSDP can work with or without its central directory service called Directory Services. When a service wants to join the network, it first sends an announcement message to notify other devices of its presence. This announcement can be multicast for all other devices to see and the service directory, if any, record the announcement. Alternatively, the advertisement can be broadcast directly to the service directory. When a client wants to discover a service, it can request it from the service directory or it can send a multicast message requesting it. The IETF Service Location Protocol (SLP) provides a flexible and scalable framework that enables devices to access information about the presence, location, and configuration of network services. SLP provides a dynamic configuration mechanism for applications on local area networks and is designed to scale from small unmanaged networks to large enterprise networks. Applications are modeled as clients that need to find servers connected to one of the available networks within an organization. In cases where many different clients and services are available, directory agents can be used, which provide a centralized repository for advertising services. Some peripheral devices, such as printers, use SLP. To remain simple and flexible, SLP is designed to support service discovery, but unlike Jini and UPnP, it does not invoke services. DNS Service Discovery (DNS-SD) is a way of using DNS to find services. It can be used with the Multicast DNS (mDNS) protocol to analyze the network. Unlike Microsoft's SSDP, DNS-SD uses DNS instead of HTTP. The DNS used by DNS-SD is mDNS, but unicast DNS can also be used. The Bluetooth Service Discovery Protocol (SDP) is specific to Bluetooth and supports searching by service class, service attributes, and service browsing when a Bluetooth client has no prior knowledge of available services in its vicinity. When a Bluetooth device is set to be discoverable, other devices can find it by sending a request. Devices listed as "undetectable" do not respond to the query. If the search is successful, a device responds by sending the service registration information,

device networks

131

retrieved from its service discovery database, to the requesting device. The requester can then use the service registration information to connect to the other device. Current SDPs are designed more for use in local area networks, e.g. B. Limit IP multicast range detection in Jini. This is insufficient for use on mobile devices that require access to wide area network services. Device service discovery tends to use a central directory server, e.g. B. UPnP, Jini, and SLP. In some environments, such as home environments However, in some environments, such as home environments, a centralized directory service cannot be relied on to be always available, as this increases costs and requires ongoing maintenance. Decentralized designs often use broadcast and multicast transmissions to locate service directories, but this can require a lot of network bandwidth and performance, an important consideration for wireless networks and mobile devices. The second main design type for decentralized SDP reduces the use of streams. These are dynamic directories that first announce their presence by transmitting multicast messages to nearby nodes. Because directories are provisioned dynamically, several can exist in the same environment (Flores-Corte's et al., 2006). Some existing SDPs require computing resources, e.g. B. Jini requires the use of JVM and RMI, which may not be present on many mobile devices. Existing service matching uses simple matching schemes and is done at the syntactic level, e.g. B. Jini attributes, Bluetooth SDP, etc.

4.5.3 OSGi The Open Services Gateway Initiative (OSGi)28 defines and promotes open specifications such as, e.g. B. a core platform specification, for the delivery of managed services in connected environments, such as homes and automobiles. OSGi's initial market was home services gateways, e.g. B. in video transmission decoders, broadband modems. These would then act as a gateway between the end user (and owner) of devices on a LAN and service providers that could be reached over the Internet and who wish to provide (i.e. sell) services for devices, such as home security. , surveillance of home health care and entertainment services (Hall and Cervantes, 2004). The OSGi Platform Core Specification defines the service framework to include a minimal component model, component management services, and a service registry. OSGi, in turn, uses the underlying Java VM (section 4.2.1.2) and operating system services. Application services are encapsulated and implemented in packages consisting of service interfaces along with their associated implementations and resources. The event-based handling mechanisms provided by the framework support installing, activating, deactivating, upgrading, and removing packages. Technically, the essence of implementing an OSGi service framework is a custom, dynamic Java class loader and service registry that can be accessed globally within a single Java virtual machine. In terms of the basic services supported, OSGI appears to be similar to the Jini and UPnP frameworks (Section 3.3.2.1), with the main difference being the more specialized capabilities of the management appliance. OSGi can also be used as a gateway to support interoperability between UPnP and Jini. You can also define a bridge between Jini and OSGi and an interface that converts UPnP services to OSGi services and vice versa (Lee et al., 2003). Allard et al. (2003) built a direct bridge between Jini and UPnP and highlighted the following differences between Jini and UPnP. Jini uses a set of attributes to search for specific services, UPnP does not. Jini is pure Java development while several languages are available for UPnP development. Jini events are very different from UPnP events, where state variable values are passed directly to clients: either a remote

28 Open Services Gateway Initiative, http://www.osgi.org, accessed January 2007. The open source implementation of OSGi is available at http://oscar-osgi.sourceforge.net or Equinox, or at http: //www. eclipse.org/equinox/, accessed January 2008.

132

Smart phones, cards and device networks

The callback mechanism or the proxy can use a thread to periodically poll the jini service for state changes. In theory, Jini, UPnP, and SLP should be easily modified to run on Bluetooth because they can run on IP networks, and because IP protocols can easily run on Bluetooth. UPnP interactions can span multiple devices and services, e.g. B. when a user wants to set up and control sessions between multiple data sources and sinks. UPnP allows streams and other session-oriented protocols to be added to specific devices or service profiles as needed. There does not appear to be a pressing need to add stronger soft-failure semantics, complex session or transaction semantics to the core UPnP architecture. Although SOAP over HTTP is very computationally inefficient,29 neither the cost of its implementation nor its performance are perceived as a significant problem.

TASKS 1. Characterize the nature of mobility in the following types of devices in terms of the mobility classification given in Section 4.2.3: WLAN transmitter, computer, phone, smart card, camera, television, and printer. 2. To simplify application and user access, mobility needs to be transparent in terms of how to locate and target mobile users, how to route data to mobile recipients. 3. Compare and contrast three types of mobile service visibility: Virtual User Environments (VUEs), Mobile Virtual Terminals (MVTs), and Virtual Resource Management (MVTs). 4. Discuss the motivation for using the mobile code. Discuss some mobile code designs based on where the code is running and who determines when the mobility occurs. Describe the security challenges and provide some solutions to address them. 5. Compare and contrast SMS, WAP and i-mode as infrastructures for mobile information services; Designs based on three-tier vs. two-tier server and thick client vs. thin client architectures, with or without client-proxy; Technological models for mobile data communication devices such as J2ME and .NET CF. 6. Why are smart cards considered privacy-invading technologies? 7. What is OSGI? Discuss how OSGI can be used in a multi-vendor device discovery environment. 8. Compare and contrast a microkernel operating system with a monolithic operating system plus middleware service model. Which one is better to use on portable mobile devices? 9. Discuss the design of an operating system for mobile use to address the increased prevalence of heterogeneous mobile access terminals; the need to route messages dynamically as the user moves, the need to deal with resource constrained devices, and the need to conserve power for a mobile terminal with a finite power supply. 10. Contour designs for low power CPU.

29 One of the challenges of using SOAP/XML/HTTP for wireless communication is that it is quite detailed and requires a lot of bandwidth. Some solutions involve ditching the XML byte encoding and compressing messages in a generic or XML-specific way, such as B. XML binary. It's often better to use SAX (handle XML events as messages are read) rather than DOM (read all messages in the tree structure into memory and then parse them). HTTP could become obsolete and be replaced by the use of persistent connections and one-way asynchronous messaging. The most efficient use of HTTP involves negotiating HTTP parameters once and using more compact protocol headers.

references

133

References Allard, J., Chinta, V., Gundala, S., and Richard G.G. (2003) Jini meets UPnP: An architecture for Jini/UPnP interoperability. In Proceedings of the Symposium on Applications and the Internet (SAINT'03), pp. 268-275. Baratto, RA, Potter, S, Su, G and Nieh, J (2004) MobiDesk: mobile virtual desktop computing. In Proceedings of MobiCom '04: 1-15. Barton, J. (2003) From the server room to the living room. ACM Tail, 1(5): 20-32. Bellavista P, Corradi A, and Stefanelli C (2001) Mobile Agent Middleware for Mobile Computing. IEEE Computers, 34(3):73-81. Brooks, R.R. (2004) Mobile code paradigms and security problems. Internet Computing IEEE, 8(3):54-59. Chan, ATS and Wan, D.K.T. (2005) Mobility web services in your pocket. In Proceedings of the IEEE International Conference on Web Services (ICWS'05), pp. 159-166. DiBona, C., Ockman, S. and Stone M. (eds.) (1999) Open Sources: Voices from the Open Source Revolution. New York: O'Reilly. Ellis, C. S. (1999) The case for high-level energy management. In Proceedings of the 7th Workshop on Hot Topics in Operating Systems, pp. 162-167. Enoki, K.I. (2001) i-mode: The Mobile Internet Service of the 21st Century. In IEEE International Solid-State Circuits Conference, ISSCC 2001, pp. 12-15. Flores-Corte's CA, Blair GS, Grace P (2006) A multi-protocol framework for ad-hoc service discovery. In Proceedings of the 4th International Workshop on Middleware for Pervasive and Ad-Hoc Computing, MPAC '06, pp. 10-16. Fuggetta, A., Picco, G.P. and Vigna, G. (1998) Understanding code mobility. Trans. IEEE. Software Development, 24(5):342-361. Goodman, D. G. (2000) Wireless Internet: Promises and Challenges. IEEE Computers, 33(7):36-41. Halle, RS & Cervantes, H. (2004) OSGi Implementation and Experience Report. In Proceedings of the 1st IEEE Consumer Communications and Networking Conference, CCNC 2004, pp. 394-399. Helal, S. (2002) Standards for service discovery and delivery. IEEE Generalized Computing: 1(3): 95-100. Husemann, D. (1999) The chip card: Don't leave home without it. Parallelism IEEE, 7(2):24-27. Johnson, S. (2006) Java in a teacup. ACM Tail, 4(3): 36-41. Lawton, G. (2001) Navigating the mobile Internet. IEEE Computers, 34(12):18-21. Lee, C., Nordstedt, D., & Helal, S. (2003) Enabling Smart Spaces with OSGi. Pervasive Computing, 2(3): 89-94. Liedtke, J. (1996). On the way to real microkernels. ACM Notices, 39(9): 70-77. Moura, J.A., Oliveira, J.M., Carrapatoso, E. and Roque, R. (2002) Service provision and resource discovery in VESPER. In Proceedings of the VHE.IEEE International Conference on Communications, ICC 2002, 4: 1991-1999. Pashtan A. (2005) User mobility and location management. In mobile web services. Cambridge: Cambridge University Press. Pillai, P. and Shin, K.G. (2001) Real-time dynamic voltage scaling for low-power embedded operating systems. In Proceedings of the 18th ACM Symposium on Principles of Operating Systems, Banff, Canada, pp. 89–102. Satyanarayanan, M. (2001) Pervasive Computing: Vision and Challenges. IEEE Personal Communications, 8:10-7. Regal, K.M. and Procaccino, J.D. (2002) Evolution of smart cards. ACM Notices 45(7): 83-88. Tao, L. (2001) Paradigm shift with the application service provider model. IEEE Computers, 34(10):32-39. Vaxevanakis, K., Zahariadis, Th., and Vogiatzis N. (2003) A review of wireless home networking technologies. ACM's Review of Mobile Computing and Communications SIGMOBILE, 7(2), 59-68. Zachary, J.M. (2003) Mobile code protection in the wild. Internet Computing IEEE, 7(2):78-82.

5 Human-Computer Interaction 5.1 Introduction The term Human-Computer Interaction (HCI) has been around since the advent of the IBM personal computer in the mid-1980s. However, the foundations of the field of HCI certainly began earlier, in the dawn of the industrial revolution. Tasks were automated and energetically assisted, primarily to save labor, but also motivated by the need to overcome some limitations in human capacity and perform tasks at lower cost. This aroused interest in studying man-machine interaction to make the interaction between them more effective. For people to effectively interact with devices to perform tasks and support activities, systems must be designed to be helpful and easy to use.

5.1.1 Summary of the chapter The chapter is structured as follows. First, HCI is considered from the perspective of the explicit (eHCI) versus the implicit (iHCI) of the interaction. Next, the variety of interfaces and interactions of ICT devices is considered, starting with the interaction of four commonly used ICT devices: desktop and laptop computers, mobile phones, and game consoles (Section 5.2). Several themes are seen to include natural interaction types such as gesture input, voice input, etc. Then, interaction modes and interfaces on a broader range of computing devices that support much more natural human-computer interaction are considered later (Section 5.3). . Next, interaction types and interfaces for human-integrated devices are considered (Section 5.4). Next, the design process for human-centered interaction that targets interactions with explicit human input is considered in relation to the Personal Visual Memories (PVM) scenario described in Section 1.1.1 (Section 5.4.5 ). This section also considers how this design process can be improved to incorporate elements of implicit HCI to complement explicit HCI. Also note that some details of HCI and iHCI are covered in other chapters of the book: tangible user interfaces based on MEMS devices and amorphous computing (Section 6.4); people context awareness to tailor ICT services to people (Chapter 7); User models based on human intelligence and artificial intelligence (Chapter 8); social and group interaction (Chapter 9); Management of smart devices in human-centric environments (Section 12.3); key challenges and prospects for HCI and HPI (Sections 13.5, 13.6).

Stefan Poslad

136

Human interaction with the computer.

5.1.2 Explicit HCI: Motivation and Properties The basic concepts of HCI are:

• Humans: single or multiple users with different physical and mental abilities that interact cooperatively or competitively;

• Computers – not just PCs, but also a variety of embedded computing devices and a variety of device sizes, such as dust, tabs, pads, and boards (Section 1.4.1);

• Interaction: can be controlled through a command or by manipulating virtual objects (window, desktop), but can also include more natural interaction, such as voice interaction, gestures, etc. HCI refers to the processes and models for the design and operational interface of some products. For many users, the user interface (UI) part of the system is the product. Explicit HCI puts the user at the center of the interactive system, so the control of the system is responsive and controlled externally by the user, rather than the system being controlled internally. Poorly designed user interfaces can result in higher training costs and usage costs, and of course, lower product sales. Reasons for higher training costs include users spending time trying to figure out what's going on, trying inappropriate computing services, and degrading task quality. Users may feel that a particular machine is forcing them to perform tasks in a way that they would rather not (without control). Users may need to relearn how to perform tasks: starting over reduces productivity. Poor user interfaces can lead to higher error rates, which are unacceptable in safety-critical systems that require some human interaction. The motivation for HCI is clear; to support more effective use (Dix et al., 2004) in three ways:

• Useful: Complete a user task that the user needs to do; • usable: do the task easily, naturally, safely (without risk of error), etc.; • used: enriches the user experience by making it attractive, attractive, entertaining, etc. The success of a product largely depends on how useful it is to the user and how useful it is for him. This combination has been clearly summarized as Heckel's law and Heckel's inverse law (Derrett, 2004). Heckel's Law states that the quality of a device's user interface is relatively unimportant to user acceptance if the perceived value of the device is high. Heckel's inverse law states that the importance of user interface design when introducing a device is inversely proportional to the perceived value of the device. These laws state that while the ease of use of the user interface is important, the primary concern is the utility of the device itself. If a complicated user interface makes it difficult to include a device, the device probably doesn't have enough perceived value to be useful.

5.1.3 Complexity of ubiquitous explicit HCI Explicit HCI is complex for UbiCom scenarios, even if well designed for individual devices, because we need to use tasks as part of activities that require access to services across multiple devices, since devices of different types of people because users participate in multiple simultaneous activities, because users participate in activities that can occur in multiple physical settings, because activities can be shared by participants, and because activities must occasionally be suspended and resumed. The amount of explicit HCI used can overwhelm users as more and more computing devices require input, even if the computers are individually designed for use. Individual devices can be

Introduction

137

easy to use but sometimes multiple devices may need to be used simultaneously to support multiple concurrent user activities e.g. B. Receiving a phone call while doing another activity. Some tasks may require the configuration of multiple individual devices and the orchestration of their individual behaviour, e.g. B. to record live video and audio content while playing pre-recorded content.

5.1.4 Implicit HCI: Motivation and Characteristics The explicit HCI (eHCI) design supports direct human intervention at reference points during normal operation of a device. Pure explicit interaction is contextless, which means users have to repeat and reconfigure the same application access every session, even if each session is repeated. It is also more about H2C (human to computer) interaction (Section 1.3.1.2), where the focus is on the human having a model of the system (a mental model) rather than the system having a model of the user. individual. For example, imagine a person entering a very dark room to retrieve an object, he or she will explicitly turn on the light. A key part of usability in interaction design is placing the switch where it is convenient to reach and possibly lighting it so it can be activated in the dark. Alternatively, human interaction can be modeled implicitly, e.g. B. Instead of requiring people to turn on a light when someone enters the room, the environment is smart. It contains devices that monitor room entry and automatically turn lights on for authorized persons, an example of implicit interaction. The shift from explicit interaction design to implicit interaction design will be a key enabler for effective ubiquitous computing systems. UbiCom systems must be designed to support explicit and implicit HCI. The concept of implicit HCI (iHCI) proposed by Schmidt (2000) was introduced in Section 1.2.3. It is defined as "an action performed by the user that is not primarily intended to interact with a computer system, but is understood by that system as input." Support for iHCI interaction has more to do with C2H (computer-to-human) interaction (Section 1.3.1.3). Implicit interaction is based on the assumption that the computer has some understanding of the user's behavior in a given situation: it is a type of context-aware system that recognizes the user (Section 5.7.2). This knowledge about the behavior of users in certain situations is then considered as an additional input for the computer to complete a task (Schmidt, 2000). Implicit interaction can allow part of the interaction to be hidden from users, making the device invisible. Implicit interaction is also introduced when we try to design systems with which we can interact in more natural ways1. For example, when we use hand gestures like clapping to turn a device on and off, we can also clap for other reasons, like expressing an emotion. Unless the context of the gesture is also defined and shared, clapping cannot be used unambiguously to imply that it is a command to turn a device on or off. There are some obvious challenges in supporting implicit interaction. Sometimes it can be difficult to accurately and reliably determine the user's context because the subject is not determined, the user has not clearly decided what to do, or because the subject's environment is not determined. Implicit interaction, in contrast to explicit interaction, reduces or even eliminates explicit user control. However, the determination of the user's context may attract and divert the attention of users to directly interact with them, or the determination may be inaccurate.

1 Natural means that, as humans, we can use our senses directly to interact with artifacts in the physical world, rather than with a tool that causes us to shift our attention from the task to the tool. However, in some cases, our familiarity and experience with a tool is so great that it seems to become a part of us, interacting through these tool interfaces also seems natural, e.g. For example, some people find that "typing text" is more natural with a keyboard than with a pen. Familiarity is cultural and subjective.

138

Human interaction with the computer.

because you can only model users indirectly, e.g. B. observing user interaction and drawing conclusions about user behavior. Systems can also take time to learn and build an accurate model of a user. Partial iHCI systems can be very useful in practice if designed accordingly. For example, they can be used to anticipate actions, prioritize decisions, and reduce user interaction and overhead.

5.2 User interfaces and interaction for four common devices Four of the most widely used networked ICT devices are personal computers in their various forms, such as desktop and laptop computers, portable mobile communication devices, game consoles, and remote-controlled AV displays. , game players and recorders. Each of these can be designed to perform a common set of functions, such as AV player, recording, output, data, and AV communication (Section 2.3.2.1), acting as a hub or portal for access to multiple services and interoperability.

5.2.1 Diversity of ICT device interaction The term computer in the acronym HCI has a much more diverse meaning in the UbiCom area. It refers to any device with a programmable IC chip inside it, including various MTOS (Multitask Operating System) devices. Common multitasking devices include desktop and laptop computers, mobile phones, game consoles, AV recorders, and players such as televisions, radios, and cameras. Each of these supports audio output and output display interfaces and keyboard input or haptic keyboard. The user interfaces of these devices are primarily visual. Keep in mind that designing a universal visual interface interaction model does not work equally well across the wide variety of types and sizes of display and input devices. Even more numerous than MTOS computers are computers embedded in devices that perform specific tasks, such as B. various home appliances, vehicle control systems, ticket machines, ATMs, building controls, etc. Their interfaces are much more diverse, and in many cases have fewer input controls, typically consisting of a series of control buttons and knobs, and outputs such as one or more LEDs (light-emitting diodes) and LCDs (displays of glass). liquid). In Section 1.4, six different form factors for ICT devices were considered. Of these, decimeter sizes of around 5 to 20 centimeters are the most common shape for ICT devices today (Figure 5.1). There are several dimensions by which devices can be characterized:

• Height: hand size, centimeter size, decimeter size vs. micro vs. body size or • • • • • • • • • •

greater than; haptic input: two-handed versus one-handed versus hands-free operation; Interaction modalities: single versus multiple; individual user versus joint interaction: in the personal area, in the friends area or in the public area; Human operator posture: lying down, sitting, standing, walking, running, etc.; Input control distance output display: centimeters to meters; Put into operation: fixed versus mobile; Connectivity: standalone vs. networked, wired vs. wireless; Tasking: single-task devices vs. multi-task devices; Access to multimedia content: oriented to voice and text communication, oriented to alphanumeric data or text, access to AV content; Embedded: embedded embedded devices versus dynamically interoperable devices.

User interfaces and interaction for four widely used devices

139

Input Type 2-hand pointer input, internal/external display 2-hand key input, internal display 1-hand key input, internal display

Game console, PDA computer, mobile phone AV player, radio

1-hand key input, external display

TV mobile phone satellite navigation.

Keyless entry, internal display Keyless entry, no display

AV Wand TV Projector

Audio receiver millicentimeter

decimeter

Metro

output size

Figure 5.1 Range of sizes of ICT devices in common use in the 2000s

The current interaction with ICT devices is dominated by the explicit interaction between humans and devices, the use of MTOS systems and interfaces with visual output.

5.2.2 Personal Computer Interface Although Windows systems were demonstrated as early as 1965, the basic idea was demonstrated as early as 1945 by Bush (1945) in his MEMEX system: "It consists of a desktop, and although it can presumably be operated at a distance ,it's mostly the piece of furniture you work on.At the top are tilted translucent screens onto which material can be projected for comfortable reading.There is a keyboard and a set of buttons and levers.Otherwise it looks like an ordinary desk "." Section 13.6 discusses changes to UIs and discusses UIs in the future. Since the early 1980s, computers have been increasingly used by non-specialist users. The keyboard and visual command interface, which allowed entering and displaying text line by line dominated computer interfaces Users had to be able to remember the name of the command and the syntax of any parameters used to qualify the command , such as B. Data files to be acted upon by the command. The command will be executed when a delimiter such as Enter is entered. Commands within an application can only be executed sequentially. It's a pretty efficient interface when you need to repeat the same command for different records using a loop. Since the early 1990s, the WIMPS interface, which supports direct manipulation of visual objects, has gained acceptance. Users can interact with computers by entering text commands, but direct interaction can be used for many tasks by activating and moving active panels called icons and menus using mouse clicks or mouse drag and drop. Such an interface is commonly known as WIMPS (Windows Widget, Icons, Menu, and Pointer) and direct keying. It is currently the dominant interface for desktop and laptop computers. The mouse as a pointing device was first demonstrated in 1965 as a replacement for the stylus. This had its first commercial success in the mid-1980s with the arrival of the Apple Macintosh (Myers et al., 1996). The WIMPS interface is associated with a desktop metaphor. Your documents refer to windowed areas of the screen. Like documents, windows can be arranged in stacks or tiles (placed side by side).

140

Human interaction with the computer.

created, dropped, moved, arranged, and modified on the screen using the pointing device (and keyboard) in a technique called Direct Manipulation Interfaces (Shneiderman, 1983), because these interfaces allow direct manipulation of visual objects. The two main advantages of the WIMPS user interface over the command user interface are that the ordering of various commands can be much more spontaneous, and the use of direct manipulation means that users do not have to remember command names. . Myers et al. they give a good overview of the history of Windows and PC interaction of the direct manipulation type. (nineteen ninety six). Shneiderman and Plaisant (2004, pp. 213–264) describe several challenges with direct manipulation: it is not necessarily an improvement for visually impaired users2; takes up screen real estate, which is more critical on low-resolution displays; the meaning of the visual representations may be unclear or misleading to certain users; Mouse pointer control and input require good hand-eye coordination and can be slow. Designs to address these challenges include reducing the number of virtual objects to be displayed, abstracting and highlighting the essence of object features, and finding and navigating between different subsets of interconnected objects that are semantically organized. Dialogues are mechanisms in which users are informed of relevant information that they must acknowledge receipt of, or request input to narrow down a request. Typically, this type of interface is presented in the form of a pop-up window called a dialog box. Query dialogs can also use a language that requires a specific syntax to restrict queries, e.g. B. the SQL query language for relational databases. Many alphanumeric data entry applications use dialog interfaces to fill in forms, e.g. B. in an information system, or for data output, z. B. a used spreadsheet. These allow applications to receive data input in a structured way, which reduces processing by a computer.

5.2.3 Mobile Phone Interfaces PC-style WIMPS interaction does not work as efficiently on mobile devices because the display area is smaller. It is inconvenient to have multiple windows open at the same time. It can be difficult to find and resize windows and icons when they are on top of each other. Screen navigation using your fingers on a touchpad or external device can be too large and cumbersome for small devices. In addition, the keyboard for user input is smaller, and there is a greater variety of input devices. Instead of using the built-in device interface, the device can be connected to various kinds of external input interfaces that are available in the area. It could become commonplace to have screens, keyboards and Internet connections running on fixed access points and allow users to connect their own mobile access device. One-handed mobile devices such as PDAs and smartphones have used a variety of interface types to overcome their resource-constrained input and output devices in the following ways (Jones & Marsden, 2006).

5.2.3.1 Handling limited key input: multi-tap, T9, fastap, softkeys, and virtual keyboard Most key input techniques introduce modes, because the number of keys is limited and the keyboard can be constantly clicked. minimum size of the key on the screen. same interface

Please note that users may have permanent visual impairments or environmental conditions, e.g. B. Bright sunlight or user activities, such as running while looking at a device, may cause temporary visual impairment for users without normal disabilities.

User interfaces and interaction for four widely used devices

141

interaction e.g. B. Pressing the same key results in different types of input actions (multi-tap keyboard). A typical number pad used on today's standard mobile devices has 12 keys. Eight of the number keys are mapped to three alphabetic characters that can be activated by multiple keystrokes, e.g. For example, for key 2, 1 touch gives 'a', 2 touches gives 'b', etc. Multiple taps can be enhanced with the introduction of a dictionary-based text recognition method known as T9, and can achieve text input speeds of up to 40 words per minute. T9 is an example of implicit interaction complementing explicit interaction: the user types the beginning of words, the system tries to autocomplete the rest of the words based on a simple text context in the corners above the other keys on the keyboard. The top one is used for alphabetic input, the bottom one for numeric input, both are as big as a business card. When multiple keys are pressed simultaneously, the system can figure out what the user was trying to type using a technique called passive chord playing. Soft keys allow the meaning of the two left and right keys at the top of the keyboard to be determined by the information on the screen; This allows the same keys to be reused to support application and task specific options. Instead of having two programmable keys, a full mini-keyboard could be displayed, a programmable keyboard, if there is enough space on the screen. Internal pointing devices such as track pads, roller pads, mini joysticks, or even keyboard arrow keys can be used to move the pointer around the screen. Screens can also be designed as touch screens, areas of which can be activated with a physical stick-like pointer, stylus, or finger (Section 5.3.4.1). Acoustic interfaces with voice recognition allow the use of voice activation to answer incoming calls and initiate outgoing calls. However, there are additional challenges for mobile users when they: work in different physical environments, such as a noisy background; Users have limited attention; Sometimes users are less able to take care of the phone when doing other activities.

5.2.3.2 Dealing with limited production There are several techniques to overcome limited production. This was a problem for many older desktop systems and not just a problem for today's mobile devices. For example, in the 1980s and 1990s, the UNIX MTOS system used a one-line text editor program, "ed", to maximize the use of a low-resolution display. Mobile devices need to be smarter at filtering information to minimize the information available to display. Mobile screens tend not to use a single window WIMPS system due to the lack of control when using a pointer to select parts of a screen on a small screen. The single window or screen can be switched to different content using a common navigation bar on each page, or hypertext links can be used to aid navigation between pages, while scrolling can be used to aid navigation within of the page to support. There are several well-known approaches to dealing with size differences between visual content size and ad size. If the content is too large, you can easily crop it, reduce the content resolution, or use a zoom interface. By zooming (in and out) along with using the scroll (up and down) and pan (side to side) controls, users can view content interactively in parts. The marking of which part of the overview is currently zoomed into another thumbnail is also useful for orientation. Another development here is a peephole display (Yee, 2003) that uses sensors to act as a tactile user interface. The sensors determine the position of the device in relation to the user. As the device moves closer to or further from the user, its position changes and its screen may update to display different information. Two other approaches to enable small formats to display large visual content are the use of MEM-type microprojectors (Section 6.4.2) or the use of organic screens that can be folded during transport and unfolded for operation.

142

Human interaction with the computer.

Audible outputs can also be helpful. This is beneficial when the user's visual focus is already activated. For example, a vehicle's navigation system can provide audible instructions when drivers are concentrating on steering the vehicle while driving. Haptic outputs can also be used as an output device. These have the advantage that touch-sensitive signals can be received more quietly and privately. For example, vibration can be used to signal incoming calls on mobile devices carried by a person. The vibrations can vary in intensity and rhythm, allowing the incoming signal, the call, to convey more information, such as the urgency of the call and the nature of the user.

5.2.4 Game console interfaces and interaction Game consoles are an important driver and can contribute to UbiCom in a number of ways. They can introduce innovative ways of interacting with physical environments into the mainstream and help drive the use of more general and augmented reality mediated environments. Bushnell3 has pointed out that computer games have often served as an incubator for many of the innovations that drive computers (Bushnell, 1996). For example, in 1971, at the dawn of computer game development, computers did not have monitors, but used paper, punched cards, and punch tape for input and output. In order to create a stock of monitors for game consoles, the tuner was removed from the televisions, stored at Atari in bins as scrap, and the remaining monitor parts saved and modified for use in computer games. Computer games became more accustomed to the idea that video content could be generated locally, unlike in the early days of television when people questioned how the television network knew what the buttons on the television did: consumers had a very strong sense of the Presence of the television station that was actually in his house. Early video games were coin-operated and contained simpler electronic logic, such as counters, since the first microprocessors had not yet been invented at the time. Atari was the largest consumer of N-channel large-scale integration (LSI) electronics in the 1970s and 1980s, driving demand for ROM and RAM memory chips, which were then manufactured in large volume at lower cost. by chip and PC enabled with cheaper. memory. Bushnell (1996) also points out that computer games have often driven a great deal of development in human-computer interaction, such as B. interactive storylines, collaborative computing, anthropomorphism, graphical user interfaces, three-dimensional graphics, and the use of interfaces that do not be keyboard or mouse. There is a prominent usage pattern in coin-operated games that drives the use of intuitive, easy-to-learn interfaces that require minimal training and instruction: slot machine interfaces without these features were used less and generated less revenue. Forster (2005) identified seven different generations of game consoles based on the technologies they used. Current seventh-generation game consoles include the Nintendo Wii, which has departed from the standard D-Pad4 interface, using instead a wall-like one-handed wireless game console interface. It contains microsensors in the form of accelerometers located in the controller and an infrared detector to detect its position in 3D space. The development and cheaper manufacturing of microsensors and actuators (Section 6.4) can lead to a much broader range of interfaces for ICT devices and opportunities to interact with them in physical and virtual environments.

Nolan Bushnell, the founder of Atari, is often credited with founding the computer game industry. The third generation gaming controller, introduced in the mid to late 1980s, introduced an 8-way D-Pad, a four-way digital cross, with two or more action buttons, and later a joystick for two-dimensional input. This became the standard game controller, replacing joysticks, paddles, and keyboards as standard game controllers, and still is more than twenty years later.

Hidden user interface through simple smart devices

143

The main focus of many types of electronic games is the interface between the person, the game, and an elaborate scoring system. This scoring system is often adjusted in such a way that as the game progresses, it becomes more and more difficult to score points, but points accumulate larger values. When done correctly, the effect is almost drug-like on players who are said to be "in the zone." If the games do not increase the player experience enough, the games will not play. Therefore, the main focus of the games is to increase the user experience. Sometimes a game's interface requires a lot of training and experience before it can be improved. The introduction of a more natural user interface like the Wii Wand can make it easier for a wider audience to interact with the system and become immersed in the virtual gaming environment.

5.2.5 Localized Remote Control: Video Devices Many body-sized or larger devices, such as early mainframes and minicomputers of the 1970s and 1980s, large televisions and AV projectors, and automobiles5. These often include more remote and automated control to reduce the amount of manual interaction that would otherwise have to take place on the device interface. Usually these devices are number pads instead of keyboards. When users need to enter text, e.g. B. for a subtitle of an AV recording, letters are then selected using the cursor keys to navigate on a virtual (on-screen) keyboard. Remote control typically requires line of sight to a screen to see feedback and status of the device being controlled. Some large devices sometimes do not have remote control, as device operation requires local manual inputs that cannot yet be robustly automated. One type of device for which there are a plethora of remote controls is the AV player recorder, be it in the form of a mobile camera, fixed set-top box player, etc. There are several more general design and technical issues here. There are a plethora of remote control devices that have overlapping functions, and in some cases it is necessary to orchestrate multiple devices around a common function, e.g. B. adjust the volume of a home entertainment center. A localized universal remote can be helpful (Section 2.3.2.3). Devices may have limited network connectivity and may not always be connected to an IP network. Device control could usefully be operated remotely, e.g. B. because someone forgot to set a timer recording, but this may cause conflicts when the device is being used by another local user who wants to perform a different AV function. If devices were universally connected to a common network, their status and utility could be easier to access and configure remotely.

5.3 Hidden user interface via simple smart devices The WIMPS-like user interface that dominates PCs is considered by many computer scientists to be intrusive in that it requires users to consciously think about how to operate a mouse pointer interface and what keys to use are pressing to use them computer. The computer itself is localized and users need to go to its location to use it. By contrast, systems that can be placed where our activities are and that directly use natural human sensations offer a less intrusive computing interface. The following sections consider some of these more natural interfaces, beginning with multimodal interaction. Some of the four common devices mentioned in Section 5.2 now support more natural input and interactions, such as touch sensitivity, gestures to control the screen, and voice input controls.

The projector itself may be quite small now, but it's the screen that users interact with and that's much bigger. Today's cars often use a remote control to lock and unlock them.

144

Human interaction with the computer.

5.3.1 Multimodal Visual Interfaces Interaction modality refers to a mode of human interaction using one of the human senses and the type of computer input. The categories of human senses are sight, touch, hearing, smell, and taste. ICT systems have modalities that correspond to some of the human senses, such as cameras (vision), input devices such as touch screens, keyboards and pointing devices (touch, haptics), microphones (hearing), and the use of various chemical sensors. and analyzers (smell and taste). Most interactive systems predominantly use a single visual mode of output interaction between a system and a human user, but this can become overwhelming for humans as the world becomes more digitally interactive and more objects may attempt to interact with the system. user simultaneously. The use of multiple sensory channels can reduce this bottleneck by increasing the available bandwidth. Human interaction is inherently multimodal in the sense that users can use multiple modes of input and output at the same time to some extent. Systems typically use multiple instances of haptic modes for input: mouse and keyboard. Human users can receive multiple inputs from other humans and systems; h hear a voice and look at someone. Users can also create multiple edits, i.e. h Gesture and speak, streamed simultaneously. The way in which the human sense inputs and controls motor outputs is multimodal. There is an additional wealth of human senses that are not adequately mimicked using today's computer input and output interfaces. The nature of the input interaction is influenced not only by the content but also by the context, that is, by the tone of voice, by the perceived posture and by the physical proximity between the sender and the receiver, by facial expressions, eye contact and body language. and by smell and touch. ICT systems such as video conferencing have the potential to allow one party to see and hear each other and offer limited support for non-verbal and multimodal communication, but as they are often designed as single-source recording, it is difficult to imagine how you can capture it. the equivalent multimodal interaction typically used by humans. Jaimes and Sebe (2005) classify modalities into two types: human senses and other ICT device modalities such as mouse and keyboard. They define various types of multimodal interfaces. Perceptual interfaces attempt to take advantage of capture (input) and rendering (output) technologies to provide interactions that are not possible with standard interfaces and common I/O devices such as keyboard, mouse, and monitor. Attentive interfaces, or iHCI, are context-aware interfaces that rely on a person's attention as the primary input. The goal of customer service interfaces is to use the information collected to assess the best way to communicate with the user. Portable interfaces include a combination of ICT devices carried by the user, typically a video camera and microphone (Section 5.4.1). His focus is on multimodal interaction that includes visual interaction. Visual modal systems are divided according to how humans can interact with the system: large body movements, gestures, and gaze. Visual interaction can be classified into command interaction and commandless interfaces, in which actions or events are used to indirectly adjust the system to the user's needs. Vision-based motion analysis systems generally have four stages: motion segmentation, object classification, tracking, and interpretation. The integration of multiple modes is complex because the detected signals have different shapes and frequencies and are merged in humans at different levels as data, features and decisions depending on what is detected or triggered. Sharma et al. (1998) present the basic theory and models for multimode fusion at these different levels. There are two main approaches. Data from each modality can be processed separately and only combined at the end. However, human use of multimodal interactions often uses individual modalities that are complementary and redundant. Therefore, to achieve human-like multimodal analysis of multiple input signals captured by different senses, the signals cannot be considered independently of each other and cannot be combined without context at the end of the intended analysis, but must instead be processed in one space. of common characteristics according to a contextual model (Jaimes and Sebe, 2005).

Hidden user interface through simple smart devices

145

5.3.2 Gesture interfaces Gestures are meaningful and expressive body movements that include physical movements of the fingers, hands, arms, head, face, or body intended to convey meaningful information about interaction with the environment. There are three main types of body gestures: hand and arm gestures, head and face gestures, and whole body gestures (Figure 5.2). Gestures can be captured with: wearable devices such as gloves or body suits; connecting sensors such as various magnetic field trackers, accelerometers, and gyroscopes to the surface of the body; through the use of cameras and artificial vision techniques. A gesture can also be perceived by the environment as an information compression technique that is transmitted to another place and later reconstructed by the recipient, e.g. B. to support telepresence and remote control (Mitra and Acharya, 2007). Gestures can also be categorized into contact gestures, e.g. B. handshake, touch screen gesture input or non-contact gestures, e.g. B. Wave at a person or camera to get attention. This type of classification also reflects 2D gestures on a flat touch-sensitive surface as opposed to 3D gestures using a handheld device that contains gyroscopes or accelerometers. Gesture recognition has a wide range of applications for UbiCom, such as: developing tools for the hearing impaired that allow very young children to interact with computers; design of passive identification techniques such as gait analysis (Section 12.3.4.1); recognize the sign language used by the deaf; Monitor people's emotional states or stress levels (as part of Affective Computing, Section 5.7.4); classify bodily behaviors such as lie detection; Navigation and/or manipulation of virtual environments (remote control and telepresence) and to enhance interaction with ICT devices such as game consoles. The first basic gestural interfaces were pen-based gestural interfaces in the mid-1960s, which were based on stylus pens and allowed pointers to move on the screen in sync with the movement of the pen. This can also emulate normal handwritten characters and can be used as a replacement for keyboard-based text input. One of the first commercial applications was the Apple Newton PDA in 1992. Since the mid-2000s, gestures have been used in various types of game consoles, mobile phones, cameras, etc.

Gesture: Rotate or flip hand Action: Rotate or flip image Navigation Options Find Movies Find Restaurant Find Cafe Find Newsstand Find Bookstore

inclination

Navigation Options Find Movie Theaters Find Restaurants Find Cafes Find Newsstands Find Bookstores

Gesture: Screen tilt Action: Menu selection moves up

Gesture: Two-finger stretch Action: Stretch image

Figure 5.2 Using Rotate, Tilt, and Stretch Gestures to Control a Screen

Human interaction with the computer.

146

Sony's EyeToy, a peripheral that connects to the PlayStation game console, is a motion-sensitive camera that can capture a person and then project them onto a virtual landscape on a video screen. Movements of the person in the physical environment can be duplicated to allow the virtual display character to selectively interact with the virtual landscape. Several current wearable devices contain sensors, such as built-in microgyroscopes, to detect the orientation of the devices to detect if they are tilting or tilting from landscape to portrait. Tiles can be used as a gesture to scroll up and down a list, e.g. B. a list of menu commands. Flip can be used to rotate a view 90 degrees from horizontal to vertical. Certain physical gestures between people can automatically trigger actions in a virtual system. For example, in a certain context6, a handshake can signal the agreement of a deal. Physical gestures can be linked to trigger exchanges and interactions between body area network or BAN devices (Section 11.6.4) and between people using inductive or capacitive electric field effects. This, in turn, can be used to enable authentications, digitally signed agreements, and distributed purchase requisitions between companies (Figure 5.3). Zimmerman (1996) first demonstrated a prototype BAN system that allows users to exchange electronic business cards by shaking hands. There are several design challenges in the use of gesture interfaces, including how the ICT system recognizes, classifies, defines, and associates gesture start and end with specific virtual gestures.

Interaction between humans and physical artifacts.

Interaction between human and virtual device.

physical person-to-person interaction

Physical person-to-person interaction triggers machine-to-machine interactions Figure 5.3 Human-to-virtual device interaction, human-to-physical device interaction, human-to-human physical interaction, which in turn can trigger human-to-device interaction

Context is critical to reducing the inherent nature of one-to-many relationships between gestures and possible actions to a one-to-one relationship between a gesture and an action. In this example, the context allows a handshake to be used as a greeting or a forced handshake to initiate a specific business transaction.

Hidden user interface through simple smart devices

147

environment objects. When someone points to multiple things in your field of vision, it may not be clear what they are pointing at. There are often many-to-one mappings from concepts to gestures and vice versa. Gestures can be static or dynamic. For example, stop using a horizontal slashing motion of the hand instead of holding the palm forward in a vertical position. Gestures can be ambiguous and are often given incompletely. Similar to speaking and handwriting, gestures for the same person can vary between different usage sessions, vary between people, and be culturally specific. Gestures can vary within an individual because spatial information varies depending on: where it occurs (spatial information); the path it takes, which may contain space-time variability (pathic information); the sign it makes (symbolic information) and the use of affective information (its emotional quality) (Mitra & Acharya, 2007). Computational approaches to deal with ambiguity, uncertainty, and variability of gestures include mathematical models based on Hidden Markov Chains (HMC) and soft computing-based tools or approaches such as fuzzy logic (Section 8.6.2 ). Most approaches first wait for the gesture to complete, then perform gesture segmentation and then recognition, resulting in a delay between performing the gesture and classifying it. An alternative technique is to try to recognize each segment of a gesture as it happens by computing a competing differential observation probability (CDOP), the difference in observation probability between the gesture and the non-gesture. There are two other features of gestures that will challenge their acceptance for ubiquitous computing. Although the gesture can be recognized, the gesture can have a one-to-many relationship with the object to which it relates, e.g. for example, pointing to objects in the physical world can refer to multiple objects within the same depth of field or to different depths of field of view. Gestures are a relatively slow input method, especially for repetitive actions, compared to direct text input7 and direct manipulation.

5.3.3 Reflective versus active displays E-books are lightweight8, thin, durable, high-performance, pocket-sized devices with touchscreens that allow you to turn pages by touch. A key difference between computer screens and eBooks is the type of screen used. E-book displays are designed more like paper, reflect light rather than transmit it, require no power to reflect light, are readable in direct sunlight, and equally viewable from any angle. The current material of choice for making electronic books is based on electrophoretic screens or EPDs (Dalisa, 1997; Inoue et al., 2002). EPDs are reflective screens that use the electrophoretic phenomenon of charged particles suspended in a solvent. Displayed text and images can be repeatedly electrically written or erased (Figure 5.4). EPDs had a serious reliability problem in early products. The performance of their displays deteriorated due to lateral drift of particles caused by gravity and agglomeration of particles caused by direct contact between particles and electrodes. Consequently, EPDs were not initially used commercially in actual devices. To solve this problem, Inoue proposed in 1987 the use of microcapsules to incorporate the electrophoretic material (Inoue et al., 2002). EPD particles are actually contained in thousands of microcapsules deposited on a substrate. Contains microcapsules

7 The average computer user can type about 30 words per minute, while the best typists can type four times that amount. Keyboard input is a faster method of entering text and issuing commands than gestures when keyboard input is not considered too intrusive. 8 They are also much lighter and more portable than analog book sets. An e-book could easily store the equivalent content of hundreds, if not thousands, of physical books.

Human interaction with the computer.

148

Transparent electrode Negatively charged white particles Microcapsule Transparent liquid Positively charged white particles Electrode

–

Figure 5.4 Electrophoretic screens are reflective screens that use the electrophoretic phenomenon of charged particles suspended in a solvent.

Negatively charged black particles and positively charged white particles suspended in a transparent liquid. When a positive electric field is applied to the bottom of the substrate, the black particles move to the top of the microcapsule, making that pixel appear black, and when a negative field is applied, the pixels move to the top, making that pixel appear black. white. Microcapsules are designed to be bistable. Once the field has been applied to attract the particles to form white or black areas on the substrate, it can be removed. Unlike active displays, power is not required to maintain black and white areas. EPD-based products focus on the concept of printing the ink rather than the "paper" substrate or screen, emphasized by the use of the term electrophoretic ink or electronic ink (Jacobson et al., 1997). Readability and legibility continue to be a critical issue for electronic books (Harrison, 2000). Additional requirements also include portability, the ability to annotate while preserving context, the ability to quickly flip through or flip through pages, cross-reference across multiple documents, and have instant accessibility with a quick launch or from sleep mode. These requirements vary based on the different types of reading required. Ihlstrom et al. (2004) discuss future design and application scenarios for electronic newspapers, ie electronic paper newspapers. Their findings were that the functional requirements for the electronic newspaper must consider mobility, interactivity, adaptation to special audiences and personalization. Various issues related to navigation, pagination, structure, and the use of overviews were also discussed during prototyping, leading to design proposals for a one-sided display. Inbar et al. they also suggest the design of future electronic messaging devices and services called epaper. (2008).

5.3.4 Combining Input and Output User Interfaces In the UIs discussed so far, input devices are separated from output devices. For example, a pointer input device, such as a mouse or trackball, emits only a single position (x, y) event at any given time along with one, two, or three key presses. The input status is only available as a visual indication. One of the first combinations of user input and output devices is the touch screen, which allows the user to touch the screen to activate an on-screen selection. Two other examples that merge input device and output user interface are tangible user interfaces and organic user interfaces. Rekimoto (2008) compares and contrasts organic and material user interfaces. Tangible UIs tend to be domain-specific, while organic UIs are more generic and less application-focused. Tangible UIs are more logical or manipulative oriented, while organic UIs are more emotional or communication oriented.

Hidden user interface through simple smart devices

149

5.3.4.1 Touch screens Touch screens are displays in which the position of contact with the screen can be detected, usually with sharp physical objects such as light pens or fingers. An event can then be raised for an associated visual at that position and an associated action can be triggered. In a way, a touch screen behaves like a two-dimensional flat smart skin. Wherever it is touched, a virtual object can be activated. A touch screen can behave as a soft control and display panel that is reprogrammable and adaptable to a variety of applications and users. Touch screens can work with a variety of electromagnetic principles, such as resistive, capacitive, surface acoustic waves, etc. Resistive touch screens consist of two thin resistive electrically conductive metal layers separated by a small gap. When an object touches the screen, the layers connect at the point of contact, and using basic electrical principles such as Kirchoff's laws, the position can be inferred from the position-dependent current flow. Touch screens are easy to use, durable, require no additional work space, and have no moving parts. These properties make them ideal for many workplaces and public areas. Touch screens were originally developed in the 1970s, but did not become widespread until the 1980s (Pickering, 1986). Touch screens are now routinely used in many applications and many devices, such as POS terminals, mobile devices and game consoles, vehicle information and navigation systems, kiosks. These devices not only support pointing, but can also support specific single finger gestures to display the next command on the screen, and support various finger gestures such as .

5.3.4.2 Tangible interfaces A tangible user interface9 (TUI) is a user interface that augments the real physical world by connecting digital information with everyday physical objects and environments. An early application was the Tangible Bits project led by Ishii, reported in 1997.10 Other project examples are Sony DataTiles and, to some extent, Ambient Wood (Section 2.2.4.4). Tangible interfaces can be realized by connecting microsensors and actuators (Section 6.4) to physical objects. These can then be used as input devices to allow their manipulation to produce data streams on an output device or a virtual view in a related virtual environment, eg being tracked in a virtual view (Section 6.2). Fishkin (2004) provides a TUI taxonomy based on embodiment and metaphor. Embodiment refers to something taking physical form, and metaphor refers to an understanding of actions in the physical realm that have equivalent actions in another virtual conceptual realm. Four types of embodiments will be described. Full realization is when the output device is the input device: the state of the output device is fully embedded in the input device, e.g. B. a robot (Section 6.7). Another example is the Claytronics project (Golstein et al., 2005), where collections of MEMS components can adjust their physical shape. A close realization means that the output occurs in the vicinity of the input object, usually directly in its vicinity. The output is tightly coupled to the focus of the input, e.g. a stylus, a computer mouse, a game console, a steering wheel, a magic wand, etc. Ambient realization means that the output is "around" the user, e.g. B. an ambientROOM device

Tangible user interfaces are also called passive real-world props, tangible user interfaces, manipulative user interfaces, and embedded user interfaces (Fishkin, 2004). 10 The use of digital physical objects, tagged with RFID, to provide a tangible user interface for ICT applications has been around much longer and preceded much Informatics-HCI research on tangible user interfaces (Section 6.2).

150

Human interaction with the computer.

uses audio and light from physical objects to refer to states in virtual views. A remote implementation is where the output is remote from the input, e.g. B. a TV remote control that switches visual attention between input (the control) and output (the TV screen). Fishkin (2004) groups metaphors into two types: those related to the shape of an object, a nominal metaphor, and those related to the movement of an object, a verbal metaphor. Systems also cannot be nouns or verbs, or have metaphors for nouns and verbs. A complete metaphor is when the physical system is the virtual system, e.g. a robot or a collection of MEM devices (Goldstein et al., 2005).

5.3.4.3 Organic interfaces Rekimoto (2008) uses the terms organic and organic interaction for such interfaces because they are more similar to natural human-human interactions, such as shaking hands and gesturing. Vertegaal and Poupyev (2008) define three characteristics that characterize organic user interfaces. First, the screen can be the input device. Second, the screen can take any shape. Third, the announcements are subject to change. According to Ishii (2008), who first proposed the concept of tangible UIs, first generation tangible UI components could not change their shape, only the spatial relationships between components. However, second generation tangible user interfaces can change form. Schwesig (2008) examines how the analog physical world can interact with the digital virtual world. One approach is to provide real physical objects with digital properties, e.g. B. Tangible UI and augmented reality. An alternative approach is to simulate physical environments in virtual environments (virtual reality). But real or simulated physics can also be very limiting: since virtual systems are not subject to the laws of the physical world, e.g. B. Hypertext links when printed on paper. Organic interface design represents a less literal approach that, instead of focusing on physical objects or metaphors, emphasizes the analogous, continuous, and transitory nature of physical reality and human experience. This potentially allows organic systems to combine the strengths of the physical and virtual or digital world: by combining responsive analog input devices with responsive graphics. User experiences can be created that take into account the complexities of physical interaction. Organic Light Emitting Diode (OLED) displays are based on organic polymer molecules that form conductive and emitting layers of the display structure, which are fused together by some form of printing (Co and Pashenkov, 2008). These are widely considered to be the successors to the ubiquitous LCD (Liquid Crystal Display) display. OLED screens have several advantages over LCD screens. OLED screens do not require backlighting (energy saving). They are visible from oblique angles. OLED displays are built from transparent layers, so the red, green, and blue layers can be stacked in such a way that a full color (RGB) pixel is a single pixel combined with full color with depth, rather than a group of red, green, and blue pixels, which might be better resolution results. These displays can also be made with flexible substrates that allow any physical object covered with these substrates to behave like a smart skin. This has many applications for new types of screens. One example is that screens can be folded or rolled to reduce the size of the physical form during transport and then expanded again when operated, realizing a more portable version of a large screen.

11 In theory, this can support the illusion of invisibility. If a car or cape is coated with this material, and many MEMS cameras could be embedded in the material, they could capture the scene behind an object and then project it onto the object's front view screen. To a viewer in front of the screen or the object, it appears as if they are looking directly at the scene behind the object, the object has vanished, a true vanishing computation.

Hidden user interface through simple smart devices

151

It is interesting to note that for the Pileus umbrella, which uses the umbrella as a shade (Section 2.3.3), OLED could have been used as the umbrella cloth, and therefore as a shade. However, due to the physical environmental requirements for using the display device in windy conditions (Section 5.5.3), OLED fabrics were deemed not durable enough to be used in severe conditions, they could tear or short circuit. and therefore OLED considered inadequate. Instead, a projection system was used to project the content onto the screen surface.

5.3.5 Auditory interfaces Auditory interfaces12 have the advantage of supporting hands-free input and output, which can be used as an additional modality of interaction when the visual senses are already being used consciously. Audible interfaces can support natural language text input as a substitute for keyboard text input. There are several key design challenges, namely interpretation of acoustic input in the presence of audio noise,13 access control for voice activation (speech recognition) and natural language processing of speech input, p. B. to recognize commands. There are two basic auditory interfaces: speech-based and non-speech-based (Shneiderman & Plaisant, 2004, pp. 374-385). Audio interfaces can provide an interface between the user and a computerized application that allows spoken interaction with the applications in a relatively natural manner compared to WIMPS interfaces. In 1969, John Pierce of Bell Labs said that automatic speech recognition would not be a reality for several decades because it required developments in artificial intelligence. However, work on the development of hidden Markov models (HMM) began in the early 1970s and this led to the launch of the first commercial speech recognition products in the 1980s. Today, their accuracy it is around 95% or better under ideal conditions. McTear (2002) gives a good overview of the problems of language systems. There are several important uses for speech recognition software. In general, discrete word recognition systems with a limited vocabulary are more accurate and more widely used than continuous speech recognition systems. Speech-based authority interfaces are used in telephony interactive voice response, or IVR, a computerized system that allows a person, typically a telephone caller, to select an option from a voice menu and interact with a computer system. Additionally, the auditory interfaces can be used in voice command, dictation, and transcription user interfaces. Along with haptic interfaces, auditory interfaces are especially important for visually impaired users. There are a variety of listening interfaces that are not voice based. Non-speech audio cues include earcons, synthetic audio sequences associated with structural context in a user interface, such as a menu hierarchy, which can also be used to indicate status during an interaction. Sounds are used in a variety of machines and devices to provide warnings and highlight abnormal behavior.

5.3.6 Natural Language Interfaces In general, an interaction is easier to process and understand when it is defined using an expressive language that has, but requires, well-defined syntax or grammar and semantics.

12 The focus here is to use audio to actively control devices and services, rather than passive audio devices that only receive and transmit audio content. 13 There is a story about the first demonstration of a well-known speech recognition system in the 1990s, which was suddenly curtailed when, while the audience was in attendance, someone called the MS-DOS system disk format command as input, which the system then ran.

152

Human interaction with the computer.

that users already know the syntax. Assuming the users were educated to read and write, the users could use the languages they already use to converse with each other to converse with the machine, i.e. the language. NLI can also be useful for casual users who don't remember system command syntax. Although NLI has these advantages, its use also has several disadvantages15 (Shneiderman and Plaisant, 2004, pp. 332-341). The semantics can be ambiguous, consider e.g. For example, the question "Did you see the man in the park with a camera?", which makes it difficult for machines to parse the query. The meaning could be clarified by more dialogs between the user and the machine, but this reduces performance. NLI can be particularly verbose and slow when you need to enter a series of commands. Because the computer is much faster at displaying information, it can also do so much faster, allowing users to select from the available information to create a query or response. Different users may also have different interpretations of the domain in terms of its scope and the meaning of the terms. It is also complex for a system to support multiple natural languages and for the system to find equivalences between them. Natural language commands can be given in three forms: they can be typed, they can be handwritten, and they can be spoken. The latter two ways require additional pre-processing to convert handwriting to text and speech to text, respectively.

5.4 Hidden user interface through wearable and implanted devices 5.4.1 Post-human technological model In the post-human model (Hayles, 1999), technology can be used to augment a person's normal conscious experience and sense of presence through space and time. There are three types of posthuman technology: companion, wearable, and implant. Companion technology is exogenous technology that accompanies it but is not directly connected to it, e.g. B. Personalized mobile devices, smart cards, smart keys, etc. (Chapter 4). Wearable technology is technology that is external and directly connected to humans, e.g. B. Headphones and wireless headsets connected to mobile phones to allow hands-free use of phones. Implants are internal body technologies. The obvious applications of implants are medical, to use various prostheses and bio-implants, to overcome paralysis in limbs and muscles, and to regulate and treat irregular biological phenomena such as cardiac activity. Future prospects for the posthuman model are also discussed in Section 13.7.1.

5.4.2 Virtual reality and augmented reality Most computers today display visual information in two dimensions, although simple three-dimensional or 3D effects can be created using shadows, object occlusion, and perspective. These are an important element of game consoles that increases user satisfaction in the main group of users. More realistic 3D effects can be created by imitating the stereoscopic vision of the eyes.

The Loebner AI Prize, http://www.loebner.net/Prizef/loebner-prize.html, accessed May 2008, is the first formal instance of a Turing test and is awarded each year to a computer identified as such. Judged with the utmost humanity. The Turing test is named after Alan Turing, who asked the question, "If a computer could think, how could we say that?" the computer could say think. This area is commonly known as natural language processing. 15 These limitations of the HMI are clear to most users because when searching in the application-specific help or through a web search engine in response to a query, many returned results seem irrelevant.

Hidden user interface on wearable and implanted devices

153

where each eye sees a slightly different perspective of the same scene. For example, a 3D headset or 3D glasses with two miniature screens, each showing the same scene from a slightly different perspective. Alternative techniques for simulating 3D are to use polarized light or to blank each eye in sync with the computer's frame rate so that each eye sees alternate images. As the head moves, sensors detect the change in viewing angle of the scene and the changing perspective of the scene is calculated and displayed. Virtual reality attempts to immerse a physical user in a 3D virtual world, while augmented reality attempts to make physical world interaction more virtual by digitally activating relevant objects in the real world. Both virtual reality and augmented reality aim to allow people to interact with more natural interaction than people use in the real world, such as using voice and gestures instead of using the keyboard interface and pc mouse. Virtual reality (VR) uses a computer simulation of a subset of the world and immerses the user in it using head-mounted displays, goggles, gloves, boots, and suits (Section 5.4.1). In augmented reality systems, electronic images are projected onto the real world, combining images from the real and virtual world. In this sense, virtual reality can be seen as a subset of augmented reality, in which there is no real world but only an artificial reality. One of the first examples of augmented reality was Sutherland's head-mounted display (1968). Similar systems are used today on types of military aircraft.

5.4.3 Laptop Interaction The essence of wearable computing is to embed computers into anything we normally use to cover or personalize our bodies. This includes clothing, jewelry, watches, glasses, teeth, ears, hats, shoes, and any other devices that we can conveniently attach to our bodies that can act like hidden computers. In a broader sense, devices can also be integrated into the environment that accompanies us, in our transportation vehicles they are extensions of portable computing. Laptops are particularly useful when computer access is required while a user's hands, voice, eyes, or attention are actively engaged in a physical environment. One of the earliest examples of portable computers was a card-sized hidden analog computer designed to predict the movement of roulette wheels (Thorp, 1998). An observer used hidden microswitches in shoes to indicate the speed of a roulette wheel. The computer would provide an octant to bet by sending musical tones through a radio to a miniature speaker hidden in an employee's ear canal. The system was successfully tested in Las Vegas in the early 1980s, but hardware problems with speaker cables prevented them from using it beyond testing. Around the same time, Steve Mann was also gaining experience with MTOS ICT devices instead of ASOS devices called WearComp and WearCam (Section 2.2.4.5), which is still ongoing. Mann (1997) specified three criteria to define portable computing.

• Eudaemonic criterion16 (in the user's personal space): The ICT device appears to be part of the user as seen by the user and the user's observers.

• Existence criterion (iHCI Control by user): ICT devices can be controlled by the user. This control need not require conscious thought or effort, but the location of the control must be such that it is within the domain of the user.

16 Named for a group of West Coast physicists known as the Eudaemons who independently developed the first portable computers with Mann in the late 1970s. One of their first applications was a shoe-like embedded computer that it allowed a roulette player to predict where the ball would land. Another early shoe gadget was the shoe phone used by Maxwell Smart in Get Smart, the 1960s spy parody series.

Human interaction with the computer.

154

• Ephemeral Criterion (Response Capacity): Interaction and operational delays are non-existent or very small. * *

Operational consistency: it is always active when used. Constancy of interaction: One or more output channels are accessible (eg, visible) to the user at all times, not just during explicit HCI.

Laptops represent a clear form of UbiCom because they can accompany the user anywhere. A more complete conceptual framework for wearable computing, called humanistic intelligence (HI), takes into account the information technology signal flow paths between the individual and the computer (Mann, 1998).

5.4.3.1 Head(s)-Up Display (HUD) Head(s)-Up Display or HUD is any type of display that presents data without blocking the user's view (Sutherland, 1968). Developed for military aviation, this technique is now used in commercial aviation and in automobiles. There are two types of HUD. In a fixed HUD, the user looks through a display element attached to the airframe or vehicle chassis, the system projects the semi-transparent image onto a transparent optical element, and the user sees the world through it (reality increased). In a head-mounted display, the system accurately monitors the user's line of sight and determines the appropriate image to present. The user wears a helmet or other headgear that is securely attached to the user's head such that the indicator does not move relative to the user's eye.

5.4.3.2 Eyetap EyeTap is a device worn in front of the eye that acts as a camera to record the scene available to the eye and acts as a screen to superimpose a computer generated image onto the original scene available to the eye (Mann and Fung, 2002). An EyeTap is similar to a HUD, but differs in that the scene available to the eye is also available to the computer projecting the HUD. This allows EyeTap to modify the computer generated scene in response to the natural scene. The EyeTap uses a beam splitter to send the same scene (at a reduced intensity) to both the eye and the camera. The camera then digitizes the reflected image of the scene and sends it to a computer. The computer processes the image and then sends it to a projector. The projector sends the image to the other side of the beam splitter so that this computer generated image is reflected back to the eye to overlay the original scene. For example, one use would be as a sports EyeTap that allows the user to track a specific player on a field and have the EyeTap display the relevant stats for that player as a floating field over the player.

5.4.3.3 Virtual retinal display (VRD) The virtual retinal display (VRD), also known as a retinal scanning display (RSD), draws a raster screen (like a television) directly on the retina of the eye (Johnston and Willey, 1995 ). The user sees something floating in front of him like a conventional screen. This is in contrast to earlier systems where an out-of-focus image was projected directly in front of the user's eye on a small "screen", usually in the shape of large sunglasses. The user focused their eyes on the background where the screen seemed to be floating. The disadvantages of these systems were: the limited area covered by the "screen"; the great weight of the small televisions that were used to project it

Hidden user interface on wearable and implanted devices

155

Display and the fact that the image only appears in focus when the user focuses at a certain "depth". The limited brightness made them useful only indoors.

5.4.3.4 Clothing as Computers Unlike HUD, EyeTap and VRD, which focus on single sensors, Clothing as Computers uses a network of wearable sensors and their data is merged to enable other types of non-visual contextual awareness. VanLaerhoven et al. (2002) reported their experience with a body-distributed sensor system that integrates dozens of body-distributed accelerometers into a single garment, most on the legs and the rest on the arms and torso. The leg accelerometers were integrated into a strap to allow testing and data collection of multiple users of different heights, while the others were attached to regular clothing with Velcro. The experiments showed that it is possible to distinguish certain activities of a user whose clothes have an integrated distributed sensor network. These activities could also include user gestures and basic events related to clothing, such as B. putting on or taking off a coat. These can be detected with reasonably high accuracy. Current sensors require rigid physical substrates to prevent delamination and mechanical incorporation of bulky switches. This drastically limits the physical shape, size, and tactile properties of objects using these sensors (Orth et al., 1998). This has led to the development of cloth-based computers and a product: the Musical Jacket, marketed by Levi in Europe. The music jacket includes an embroidered cloth keyboard made of a BUS of sewn metallic conductive fabric and non-conductive cotton-nylon tulle (the insulating layers), a rechargeable battery, a pair of standard speakers, and a miniature MIDI synthesizer. When the cloth button is played, it communicates via the cloth BUS with the MIDI synthesizer, which produces notes. The synthesizer also sends audio to the speakers via the fabric BUS. The power from the batteries is also distributed by the fabric BUS.

5.4.4 Computer implants and brain-computer interfaces The opposite of using computers externally is connecting them more directly to the body. Many people routinely use implants, for example pacemakers are used to regulate the electrical activity of the heart, artificial limbs can increase mobility, and contact lenses that are placed in the eye can improve contrast with trackballs. during exercise, even for people with good vision. Of particular interest is the development of devices that can adapt to signals from the human nervous system. By connecting electronic circuits directly to the human nervous system, the physiological signals that represent our thoughts, emotions, and feelings can be input directly into computers, allowing humans to operate machines through thought (Warwick, 1999). This represents the last natural interface, mind control instead of motor control of the devices. In contrast to human-machine interfaces or human-computer interfaces, brain-computer interfaces (BCIs)17 or brain-machine interfaces (BMIs) are indirect interfaces of the human brain via human actuators, i. h Haptic interfaces and machine sensors support direct functional interfaces between brains and machines, such as computers and robotic limbs. There are several design options here: the human-device interface could be positioned to have a direct or indirect connection to the nervous system, and the device could be located in the brain or

The term brain-computer interface was first coined by Jacques Vidal in the 1970s.

156

Human interaction with the computer.

located elsewhere in the body. Lebedev and Nicolelis (2006) classify BMIs according to whether or not they use invasive versus non-invasive neural signals.18 Invasive techniques, in which electrodes are implanted cranially, can record from one or several sites, and within those sites, signals of Sample small groups of neurons or larger groups of hundreds of neurons. He discusses noninvasive brain BMI first, then noninvasive brain BMI, and then invasive brain BMI. Warwick et al. (2003) reported that in 2002 an array of 100 electrodes was surgically implanted into the median nerve fibers of his left arm. A series of experiments were carried out which showed that he could control an electric wheelchair and an intelligent artificial hand and create artificial sensations in them by stimulating individual electrodes. Noninvasive BMI was first described by Vidal (1973), whose early work laid much of the foundation for the collection and computer processing of high-quality EEG signals. Characteristics are extracted from the signals, and a translation process converts these characteristics into symbols or commands to control electrical devices. Non-invasive brain BMI is more mature and has fewer safety concerns than invasive brain BMI and is considered more suitable for use in daily activities. A typical portable computer system for non-invasive cerebral BCI consists of an EEG recording cap and a body area network (Section 11.6.4) to transmit the EEG signals to a mobile device that also acts as a gateway between the BAN and Internet functions. . Navarro (2004) discusses the use of BCI to accurately predict triggers for brain activity in a highly changing environment, as is often the case when performing activities of daily living. For example, there are differences between the real environment and experimental settings when it comes to identifying and combining brain patterns to trigger certain things. BCI experiments were also conducted in virtual reality (VR) environments to more closely replicate situations where a BCI was activated. Such experiments can minimize the uncertainty of brain inputs derived from the "external" environment by being able to recreate a more extensive, identical multi-sensory experience for the user. This potentially engages the user's cognition in a broader way, resulting in similar mental states and brain activity patterns when an identifiable event occurs that triggers the BCI. Invasive BMI was first established in experiments by Chapin et al. (1999). He showed that rats could use their motor cortex to control the movement of a robotic arm to dispense drinking water. Research was launched to see if this approach could help restore motor skills in people with physical disabilities using artificial limbs or prevent neural network failures in humans by functionally stimulating paralyzed muscles. Lebedev and Nicolelis (2006) envision neuroprosthetic developments for invasive systems that could emerge in the next ten to twenty years. Invasive Brain BMI will use a fully implantable recording system that wirelessly transmits multiple streams of electrical signals originating from thousands of neurons. BMIs will be able to decode spatial and temporal features of movement and intermittent periods of immobility. BMIs will be able to use a combination of higher-order motor commands to control a multi-degree-of-freedom artificial actuator or directly stimulate multiple peripheral nerves and muscles via implantable stimulators. Much research is needed before any of these types of BMI, non-invasive and invasive, can be used more routinely like the UbiCom system to improve convenience of use accuracy.

18 Non-invasive systems that mainly use electroencephalograms (EEG) to control computer cursors or other devices such as wheelchairs have the advantage of not exposing the patient to the risks of brain surgery, but use signal channels with a capacity rate very limited, typically around 5-25 bits per second per source. Gamma-type EEGs, which represent specific cognitive and motor functions, have the highest frequency range, from 25 Hz to 100 Hz.

Human Centered Design (HCD)

157

Robustness and sustainable use of the systems and alleviation of implant-related safety concerns for the patient in relation to the psychological and physiological effects of the use of BMI systems.

5.4.5 Sense of Presence and Telepresence The posthuman model refers to a discussion of alternative realities that attempts to extend the experience of the sense of conscious presence here and now of humans located in the physical world to experiences of being elsewhere, perhaps in each other. time. A sense of presence in the experience gives the person feedback on the state of their activity. The subject perceives any variation in the sensation of presence and adjusts its activity accordingly. Human beings can experience alternate realities depending on the type of environment people are in and their perception of the environment. Reality can be technologically mediated, chemically mediated, and psychologically mediated. IJsselsteijn and Riva (2003) offer a discussion on the experience of presence in mediated environments. Non-interactive AV media, such as movies, theater, and music, and interactive AV media, such as electronic games, and even the senses of smell and touch, possibly related to previous experiences, can transport the viewer to a different realm. or perceptive. state of sense of presence from one's immediate physical environment to another distant presence or experience. Hyperreality, for example, characterizes the conscious mind's inability to distinguish between reality and non-reality such as fantasy, particularly when one is immersed in the experience. A sense of immersion can often be achieved when playing physical or electronic games, where someone is caught up and embraced by the AV medium and transported to another realm or state of perception. In this type of immersion, a person is influenced by the environment on multiple levels: perceptual, sensory, psychological, and emotional. Telepresence allows a person in one place to feel as if they are in a distant place, appearing to be present and acting in a place other than their true location. Telecontrol refers to the ability of a person in a place to control objects remotely. These can be linked to local detection of body movements, which can then be used to remotely control such objects in conjunction with telepresence. For example, it allows people to handle hazardous substances in a remote environment from a safe environment as if they were present. Wilson and Shafer (2003) describe the development of a magic wand-like device in which the user points to the device to be controlled and then uses simple gestures or speech to control the device. The intelligent environment system interprets the user's manipulation of the wand to determine an appropriate action in the context. For example, the user can turn on a light in the room by pointing the wand at the light and pressing the button. Alternatively, the user can point the wand at the light and say "turn on". Kim and Kim (2006) also developed a gesture recognition method for opening and closing curtains and turning lights on and off in a smart home environment, captured with three CCD cameras positioned at 0, 45, and 45 degree angles.

5.5 Human Centered Design (HCD) Traditional system design focuses on designing the core of the system to support action sets or service functions to handle service requests. In contrast, user-centered design, also known as user-centered design, focuses on designing the part of the system that human users interact with, the user interface. For many users, the user interface represents the product. Whereas traditional system design tends to involve users only at the beginning and end of the system development life cycle, human-centered design involves users throughout the entire design life cycle.

Human interaction with the computer.

158

5.5.1 User-Centered Design Life Cycle The basic development phases of user-centered interactive system design are similar to those of conventional functional system design (Figure 5.5). This development cycle includes four development phases: (1) requirements gathering and analysis; (2) modeling and design; (3) implementation; and (4) tests. Interaction design, or user-centered design, is closer to the prototyping cycle model. Prece et al. (2006), Dix et al. (2004) and Shneiderman and Plaisant (2004) provide comprehensive advice and guidance on human-centered design, human-computer interaction, and user interface design. The user-centered design process for interactive systems according to ISO 1340719 establishes four design principles: (1) active participation of users and a clear understanding of user and task requirements; (2) an appropriate role assignment between users and technology based on the relative competence of technology and people; (3) Iteration is inevitable because designers rarely get it right the first time; and (4) a multidisciplinary approach to design. The human-centered design life cycle includes four main groups of activities (Figure 5.5): 1. Defining the context of use in terms of scenarios, use cases, task models and the context of use of the ICT, physical and social environment. 2. The requirements of the interested parties and the organization should be specified20.

need for new products

Identify interactive design needs

requirements analysis

Understand and specify the context of use.

Design

To implement

Validate designs against requirements

The system meets the requirements

Identify the needs of stakeholders and organizations.

Validate Produce Design Solutions Final Product

Functional system design

Human-Centered Design

Figure 5.5 Comparison of a conventional functional systems design approach with a human-centered design approach

ISO 13407:1999 Human-Centered Design Processes for Interactive Systems, available at http://www.iso.org/iso/iso_catalogue.htm, accessed January 2008. 20 Requirements are defined as clear, specific statements and clear on what the system should do or how it should work.

Human Centered Design (HCD)

159

3. It is necessary to create several alternative user interface designs. Designers can suggest alternatives to help the user "go outside the box" to identify better alternatives instead of sticking with currently known designs. Alternatives are generated through research and synthesis, creativity and the search for inspiration in similar products. 4. Designs must be validated against user requirements. Harper et al. (2007) propose extending the four-phase interactive development life cycle from Study, Design, Build, Evaluate to Understand, Study, Design, Build, Evaluate. While the understand phase provides a framework for designing and exploring the human values that should be supported by a system, the study phase involves working out the details of how individuals and social groups pursue and achieve those particular endeavors.

5.5.2 Methods for collecting user information and creating models used The basic techniques used to collect user information for formative evaluation of designs can also be used throughout the user-centered design life cycle, from initial requirements gathering to user behavior modeling. The choice of appropriate user input techniques depends on a variety of factors, such as: B. different subtypes of environmental requirements, cost and time constraints, and the availability of different types of users. There are two basic types of techniques for collecting user input, depending on the type of human environment in which the user input is collected. In field studies or ethnographic studies, user feedback may be collected in the users' natural environment. Usability testing allows users to be tested in controlled environments, often in a usability lab, where cameras and input filters can observe what users do and input into a system. There are two basic ways to evaluate whether or not direct interaction or indirect interaction with users is used. These can be combined. Field and laboratory studies may involve direct or indirect user interaction. For example, use of the labeling techniques highlighted in Section 6.2 may allow tracking in the field of the changing spatiotemporal context of subjects, including humans and animals. Direct interaction techniques include questionnaires, interviews, inspections and focus groups, and cognitive walkthroughs. There are different types of questionnaires and interviews. Unstructured interviews are not scripted. These are rich but may not be easily replicated. This is a way to get informal feedback from users. This is also called quick and dirty feedback. Structured interviews are strictly written, often like a questionnaire, reproducible but may not be rich. Semi-structured interviews are scripted, but topics of interest can be explored in depth. These can provide a good balance between richness and reproducibility. Focus groups or group interviews are groups led by an interviewer who facilitate group discussions on specific topics. Heuristic inspection or evaluation: Experts and expert users inspect the user interface and are asked to test and document how a set of usability heuristics is applied to perform certain user tasks. There are different types of usability heuristics that are used to evaluate different types of interactive products, such as online communications, websites, desktop computers, and mobile devices. Cognitive walkthrough: This is an alternative approach to inspection where designers and raters, as individuals or groups, walk through the sequences of a task and document whether a user will know how to complete a task, what actions are available, whether users can interpret the system response to an action. User tracking is the main technique used to collect indirect information about users. The same tools that are used to observe users in controlled environments (in the lab) can be used in uncontrolled environments (in the field), but the way they are used is different. in it

Human interaction with the computer.

160

lab, details of user behavior can be recorded using stationary observation devices, e.g. B. Key click analysis can be performed to track users in the lab. In the field, we can focus more on user context but collect less detailed information. Keep in mind that observing user tasks makes it difficult to know what users intend to do. Therefore, observations can be supplemented by asking users to take a cognitive walk and think aloud. Predictive models provide ways to collect user feedback on a product or design without necessarily involving users directly. Quantitative models of user interaction sequences, such as clicking a button, moving the mouse pointer, thinking, and moving between mouse and keyboard, can be used to calculate task efficiency, e.g. B. how long it takes to complete a task and quantitatively compare user interactions. is the most efficient method. An example of such a method is the GOMS-KM model (Raskin, 2000). This may seem less expensive than methods that require user input, but its usefulness is limited to systems with routine or predictable tasks.

5.5.3 Defining the context of use of the virtual and physical environment In the PVM scenario, the context of use of virtual computing or ICT has an internal and an external context (Figure 5.6). Internal use refers to internal storage capacity for playback and review of recordings, etc. The external ICT context requires the AV recorder to share content with external systems to view, archive and retrieve recordings. The context of the physical environment relates to the operating conditions of the physical environment for the system).

5.5.4 Defining the context of use and human environment requirements 5.5.4.1 User properties The context of the human environment refers to user properties, social context, and usability context. The meaning of the user identifiers is illustrated by examples. In the PVM scenario, users must be able to hold the camera securely to operate basic functions intuitively.

environmental requirements

Social usability

Functional Tasks Non-functional System Spec.

Human

System UbiComp Internal: memory, display. Etc

External: Storage, QoS, Network. etc ICT

TIC

Virtually

Username

Virtual physical operating context: dark versus light conditions, etc. TIC

Physically

Figure 5.6 Interactive Design Requirements considers a broader set of requirements that go beyond functional and non-functional requirements.

Human Centered Design (HCD)

161

camera etc A configuration or set of user preferences can be specified for using a device or service, called a user profile21. On many current devices, such user profiles are device-specific and located within the device. Sometimes it is useful to be able to define and share profiles between multiple applications and devices. However, most consumer devices currently do not support shareable cross-application user profiles. Other types of user profiles, such as personas, can be used to characterize imaginary user types. Stereotyped users are similar to people, but are user types derived from collecting, analyzing, and aggregating real-world user interactions. In the PVM scenario, ICT devices are deployed in personal social and public spaces. People can also act as part of organizations and therefore the organizational context can limit the individual context of use. It may be necessary to control the device remotely so that the owner of the AV recorder can also appear as part of the scene. The camera can signal people in the scene that you are about to take a photo so that people can pose for the photo, and can indicate that the capture is complete. The AV recorder could be designed as a hidden recorder to record people acting natural instead of posing for the recording. The human context of use can also determine how the user is instructed and trained to operate the device and resolve operational problems. According to the ISO 9241-1122 usability guide standard, usability is defined as the extent to which a product can be used by specified users to achieve specified objectives with effectiveness, efficiency (time to complete a task), and satisfaction in a specific context to achieve useful goals. . Usability is not a single, one-dimensional property of a user interface. Usability is a combination of factors. ISO 940-11 explicitly mentions effectiveness, efficiency and satisfaction. However, these can often be further expanded. For example, effectiveness is often expressed in terms of learnability (ease of learning), retention (ease of remembering to use a previously learned user interface), and error rate (the number of errors per unit of time while operating a user interface to complete a task). task). . User satisfaction can also be expanded to become part of a broader set of user experience criteria (Preece et al., 2006). Certain types of UI components also have a more specific user experience. For example, dialogue systems must have the qualities mentioned in ISO 9241-11023, such as suitability for the task; self description; controllability; conformance to user expectations; Fault tolerance; suitability for individualization; and learning ability.

5.5.5 Interaction design Interaction design focuses on designing systems in such a way that they can be used by their human users. It focuses on the design of the user interface part of the system. Two main differences from traditional design are, first, that multiple alternative prototype designs can be produced, and second, that there can be (formative) evaluation of these prototype designs. The overall design of a user interface generally projects a simpler, higher-level view of the system, called the conceptual user interface model.

21 The term user profile is complex. This can mean the user's preferences for access to the device or service, or represent the configuration of a resource that is being used. User profiles may or may not be specific to the device or service. Devices can support one or more user profiles. 22 ISO 9241-11:1998 Guidance on usability, available at http://www.iso.org/iso/iso_catalogue.htm, accessed January 2008. 23 ISO 9241-110:2006 Ergonomics of the interaction between the human system and the system - Part 110: Principles of dialogue, available at http://www.iso.org/iso/iso_catalogue.htm, accessed January 2008.

162

Human interaction with the computer.

5.5.5.1 Conceptual models and mental models There are an astonishing amount of everyday things, Norman (1988) states that there are between twenty and thirty thousand of them. In addition, there are many of them, such as B. a camera, in turn from other different parts. It seems like a great challenge for humans to learn to operate and understand tens of thousands of these devices of varying complexity when the interaction with each one of them is unique. This complexity of interacting with new machines is reduced because, through experience, humans tend to build mental models that predict the behavior of different types of objects and base their interactions with familiar and unfamiliar objects on them. The user mental model develops when a person begins to learn how to use an unfamiliar system. Once the user becomes familiar with how the system works, the mental model is not used and habits develop that associate the user's actions with sensory information without thinking. The complexity of interacting with new systems is also reduced when they contain parts that provide strong clues about how they work. These are called benefits (Norman, 1988). For example, a camera lens has a rotating ring to adjust the degree of zoom. The usability of the offers is partly based on experience, it is subjective and cultural. Imagine that you are in a foreign country and you need to access a machine that has a slot as an input. It may not be obvious what to insert into the slot, e.g. a token owned by that machine or a local currency denomination, unless the slot is clearly labeled in a language you understand. Some mechanical or electromechanical offerings can invite trial and error interactions, but this can permanently damage the device or be dangerous. System designers can also aim to design interactive systems to support appropriate conceptual models. A conceptual model is an abstraction of a system or service at a level that is understandable to a user and corresponds to the user's mental model of how they think the system works. A common way to define a conceptual model (virtual objects) in a human-computer interface is to associate the virtual objects or widgets it contains with related physical world objects in a specific application domain to increase their flexibility and functionality. For example, a recycle bin placed on the virtual desktop makes it easy to discard information. However, a container is not typically placed on a physical desktop in the physical world environment. Furthermore, it is difficult to emulate the continuity of the analog world in the digital world. In the analog world, we don't close all open files if we want to continue working on them, but we have to do this in the digital world when we need to shut down the system (unless we're using system sleep or hibernate mode). Usability is diminished when a system projects a conceptual model from which a user cannot create a mental model that they understand. For example, when systems crash, low-level error code messages and help may be displayed that are relevant only to the system developer, but not to the user. Usability is also reduced when users have constructed the wrong mental model of the conceptual model of a system. There is not necessarily a single generic conceptual model that is suitable for all applications.

5.5.6 Evaluation There is a strong motivation to evaluate products during development, formative evaluation. Allows for issues to be resolved before a product ships. Allows developers to focus

24 Note that this mapping from virtual object manipulation to physical object manipulation is the opposite of the mapping used in tangible user interfaces that associate physical object manipulation with virtual object manipulation.

User models: acquisition and representation

163

to real problems that have evidence of use, not problems that you perceive or imagine but may never encounter. The business case for formative assessment is just as strong, time to market can be greatly reduced, resulting in significant cost savings. After the final release, the sales department will have a rock-solid design that they can sell without worrying about how it will perform in a future release. Finished product evaluation, known as summative evaluation, can be used to verify and demonstrate that a system meets regulatory requirements, such as security and physical environment requirements. It is necessary to plan the evaluations and pay special attention to the different types of users who will carry out the evaluations. Methods for collecting user information for evaluations may be based on those used to collect user requests. (Section 5.5.2).

5.6 User models: capture and representation User context models can be viewed from two perspectives: system user models and user system models.25 Users have a (mental) model of the UbiCom system, from how the user understand that the system works until your eHCI Interaction to make it easier (Section 5.5.5.1). The user modeling focus in this section is how UbiCom Systems can use a model of the user (the user context) to facilitate iHCI's interaction with it. The focus of user modeling for UbiCom is how UbiCom systems can capture models of the user. There are several different design options for user modeling.

• Implicit versus explicit models: implicit or indirect capture of the user model, p. B. observing the user in the face of explicit or direct user interaction.

• Modeling of (individual) user instances versus (stereo) type modeling of users: stereotypes are derived

•

• •

B. by data extraction and grouping and classification of past interactions of many individuals into groups or by hypotheses about types of users by experts (Han and Kamber, 2006, pp. 285, 382). Individuals can then be matched to their stereotypes based on some initial interactions matching those of the groups. Static vs. Dynamic User Models – Static user models are often captured in a single shot model. Some user characteristics are often immutable once created, such as: B. fingerprints, gender, place of birth of a person, while other user characteristics vary with respect to time, place, and context, such as. mood and user experience, etc. Dynamic user models vary by user task, time, and/or space, and may need to be updated from time to time. Machine learning algorithms (Chapter 8) allow a user model to semi-automatically or automatically adapt to a changing environmental context. In the case of complex user models, these can also be captured in multi-shot mode. Generic models versus application-specific models: the latter applies to a specific task or application domain, while the former can be used across tasks and application domains. Content-Based User Models vs. Collaborative Models: Content-Based User Models rely on prior content characterizations and then tailor them to an individual user's preferences for characteristic content. Collaborative user models depend on matching the preferences of individual users with the preferences of a stereotypical user and then using the latter to complement the preferences of the individual.

25 If there is bidirectional modeling of the UbiCom system, where the UbiCom system models the user and the user forms a mental model of the system's operation, this increases the complexity of the model and may introduce cyclical instabilities.

164

Human interaction with the computer.

A major problem in user modeling is acquiring knowledge about the user. There are two main ways to do this. Systems can ask the user explicitly about such interests (explicit feedback) or observe a user's use of the system (implicit feedback) and derive and anticipate certain behavior from this. Often these can be combined, either different subtypes of implicit or explicit models, or explicit models can be combined with implicit models to improve the user model. Hybrid user models can also be used. Stereotyped user models can be used in conjunction with explicit user modeling to try to reduce individual user values by trying to suggest default values from matching user stereotypes, see Section 5.6.3. Stereotypical user models are also commonly used in conjunction with collaborative user models. An important consideration is to define what to model and how to represent it. Human reasoning is predominantly qualitative or soft, e.g. B. to rate how much someone likes an article, e.g. B. Using a five-point Likert scale: a lot, a lot, good, a little, and not at all. Therefore, soft computing techniques such as fuzzy logic, probability theory, neural networks, and genetic algorithms seem well suited for this representation (Karray and De Silva, 2004; Section 8.6).

5.6.1 Indirect user input and modeling User models can be built by collecting and analyzing indirect user input and linking it to user goals related to user tasks and activities: context-aware models. Context-aware user models can take into account the context and interaction of the current user, or also include the history of previous interactions. This knowledge or model of user interactions can be used to anticipate, facilitate, and simplify interactions (anticipatory computing). It can be used to personalize user input, to filter user content and service choices (personalization). Group behavior models can be used to facilitate individual interaction (recommender systems, Section 9.4.2). Indirect user models can be improved by combining multiple contextual values, such as location, entity, activity, and time, and can produce a more complete understanding of the current situation. These user contexts can also serve as indexes to other sources of context information. For example, if the app knows the current location and time along with the user's calendar, it will have a pretty good idea of the user's current social status, e.g. For example, sitting in a meeting, in a class, waiting at the airport, etc.

5.6.2 Modeling and direct user input The primary design option for explicit user modeling is to use single shot feedback (static or periodic). With single shot feedback, a single complete interview or questionnaire is conducted. However, full interviews are often time consuming and users may not know the best way to complete a long form and may do so incompletely. The other main disadvantage of a user model is that it can become obsolete. Periodic user feedback may use fragmented dialog boxes. These are less time consuming and may fit better with session based activities. Additionally, the system can target subsequent dialog sessions by asking more specific questions in a specific context to improve the accuracy of the user model. Explicitly prompting the user would be more accurate, but it interferes with the user's current task and can be time consuming and annoying. Additionally, users may be reluctant to fill out preference forms, and again, small screens for mobile devices are a limitation. In contrast, indirect modeling of users may be inaccurate, but it does not annoy or annoy users. However, there is a potential lack of user control and loss of privacy.

User models: acquisition and representation

165

5.6.3 User stereotypes The disadvantages of asking the user explicitly can be offset by assuming an initial interest profile of the user from a stereotype, i. h the user is asked few questions, such as classifieds. Typical interest profiles for such stereotypes need to be identified in empirical studies. Subsequent user interaction with the system can modify and correct this initial profile. Stereotypes derive a user model from a small set of facts using a much larger set of facts from a user model (Rich, 1999). A stereotype is a collection of user attribute value types with a belief in the value and a set of reasons why the value is assumed. A stereotype is triggered by a user's interactions and, in turn, can predict values with other user attributes. To some extent, collaborative recommender systems use a user stereotype. Stereotype design challenges include creating new stereotypes, determining trust values, dealing with users who may not fit a stereotype, and handling excessive or incomplete input.

5.6.4 Modeling user goals and planned tasks Users typically interact intentionally with a system based on tasks to achieve a specific goal. There are several ways to analyze and model user tasks. Hierarchical Task Analysis, or HTA, is a technique for decomposing a user's goal into a hierarchy of actions. The actions are ordered, starting from the left in Figure 5.7, to follow an order to achieve the goal. Complex actions can be broken down

Plan 0: Mach 1..2; Repeat 3..7 until no more records are needed or no power, then do 8

0: record a scene of the physical world

1: turn on the camera

2: Set the camera task mode

3: Select scene

4: Set up camera recording of the scene

5: Compose scene

6: Record scene

Plan 1: Repeat 3.1, 3.2 until satisfied, do 3.3 3.1: Go to scene

3.2: Show scene

3.3: Correct scene 4.1: Adjust zoom

4.2.1: Set Flash

4.2.2: set under exposure

7: Check the recording

8: Turn off the camera

4.2: Set Plan 2: Repeat 4.1 Lighting until satisfied, do 4.2 Correction

4.2.3: Set Dodge

Plan 3: Haz 4.2.1 o 4.2.2 o 4.2.3

Figure 5.7 A hierarchical task analysis (HTA) model for part of the physical world scene recorded from the PVM scenario in Section 1.1.1. Note: Parts of the scene have been omitted for simplicity.

166

Human interaction with the computer.

in a set of simple actions to define actions at a level of detail sufficient for the system to support them. Plans can be defined to express action options and iterations. Consider the PVM scenario; There are several key user tasks that the system must support: recording a scene from the physical world, annotating a recorded scene, retrieving selected scenes based on defined search criteria, searching for recorded scenes. An HTA model for recording a physical world scene task is shown in Figure 5.7. This is a partial HTA model for the task. Camera focus, face detection, face detection tasks, etc. are not specified. This is a simplified type of scheduling model in which tasks are completely ordered, each task is treated as independent of other tasks, only single tasks can be performed, not multiple tasks, plans are static, etc.

5.6.5 Multi-User Tasks and Activity-Based Computing Much of what is called personal computing is more appropriate for office workers who work on single, fixed tasks relatively continuously for long periods of time. Conversely, there are other types of human activities that tend to be disruptive, are nomadic, are short-lived, and are likely to nest multiple user activities to achieve multiple user goals. A utility function based on multiple value attributes can be used to allow multiple targets to be weighted and combined. In the Classroom 2000 project, Abowd and Mynatt (2000) outlined several design principles for everyday computers to support more informal everyday activities.

• User activities rarely have a clear beginning or end: information from the past is often recycled,

•

B. address book, event calendar, to-do list, etc. The activities are based on previous experiences. Principles such as visibility of the current state, freedom of dialogue, and general simplicity of features play a prominent role in activity support design. Interruptions are expected: Users should be aware of incomplete actions. Also, the resumption of an activity should not start at a single point, but is related to the state before the interruption. The resumption may be opportunistic depending on the availability of others or the recent arrival of the necessary information. The interaction should be modeled as a plan that can be suspended at a certain time and resumed at a later time. Multiple activities run concurrently: Activities are continuous and context switching between multiple activities must be supported. To accommodate background awareness, interfaces must support multiple levels of "intrusion" in the communication of surveillance information appropriate to the relative urgency and importance of the events. Contexts like time are useful for filtering and matching: however, contexts like time are rarely represented in computer interfaces. Many current and future personal actions, social interactions, and decisions are based on the result, or lack thereof, of past interactions and events, e.g. B. If the application's help information searches have not returned any useful information in the past, the user should not attempt to request help at any subsequent time. Associative Information Models Needed: Information models for activities are inherently associative, as information is often reused multiple times from multiple perspectives. Associative and context-rich models of organizational support activities by allowing the user to retrieve and reinforce the validity of information from multiple points of view. These views also support the need for users to continue an activity in different ways.

Bardra et al. (2006) characterize activity-based computing as: application-independent, occurring across multiple application tasks, supporting suspend and resume, supporting user roaming, adapting to available resources, shared across multiple workers, and is context sensitive.

iHCI Design

167

In user-centric services, ICT events and service reconfigurations can be expressed in terms of various knowledge points, e.g. B. using the mental model of different users. Most TIC events are either too low or too detailed for users to understand and handle, or too high, including some uncertainty about the cause. TIC events are often handled locally within the application code. You cannot go to multiple points, e.g. B. The time is set manually on the control panel of each device instead of being set automatically by the network. Extensive user-focused knowledge portals could be created and maintained, e.g. B. Allow users to post problem descriptions in user terms instead of selecting from predefined lists of vendor problem views; These can be used to advise users and provide self-help to instruct users remotely.

5.6.6 Situational versus planned action models There are two basic approaches to task design: planned action, which involves replanning, and situated action (Suchman, 1987, pp. 49-67). In planned actions, user activities are driven by specifying user goals or goals and plans to achieve that goal (pre-planning), e.g. B. HTA (Section 5.6.4). Monitoring and situational awareness during plan execution can provide feedback or constraints to potentially change plans and make them more resilient. This approach assumes that the user is capable of defining a goal, that predetermined plans to achieve the goal can be devised in advance, and that contingency plans to adapt to significant changes can be specified in advance. An alternative approach is to set a goal but not to pre-define a plan to achieve that goal, simply to assess locally at each stage what is the best course of action that gets closer to the goal, e.g. B. actions situated to evaluate at each step what is the next optimal local action that brings the user closer to his goal. In addition, feedback on the status of the activity must reach the user so that the user can decide if he can influence a reconfiguration of the activities. Riva (2005) specifies two main approaches to model situated actions. The first approach, a cognitive-situational approach, builds symbolic models of the relationships between subjects and the properties of specific environments (supplies and constraints). This does not explain how the choice of possible actions is restricted by more than the situation itself. In addition, the social space influences the subject's activities. The second approach, an interactive approach like that of the Wenger Community of Practice, states that these are developed through joint ventures as understood and continually negotiated by their members, which work through mutual commitment that unites members as a social unit, and that they share a repository of community resources, including vocabulary and guidelines. A key concern is how a user maintains control over her planned activities and how she can potentially affect the impact of a recognized situation or a changed situation. This can be modeled by a theory of presence that seeks to distinguish between internal and external states and to experience distal attribution by perceiving an external space outside of our limits (Riva, 2005). System designs for planned actions are discussed in Section 8.7.

5.7 iHCI design 5.7.1 Characteristics of the iHCI model An iHCI design model is characterized by the following characteristics:

• Natural (human-computer) interaction can use a wide variety of physical artifacts that are found throughout the physical world and are implicitly connected to virtual computer artifact interactions. Users do not need to be aware of the presence of interfering computers at specific locations in the physical environment to access digital services.

Human interaction with the computer.

168

• User models can be used to anticipate user behavior based on: *

Models of past individual user interactions that can be used to anticipate future user behavior; Individual user interaction models can be grouped into user stereotypes to predict user behavior based on the group to which the person belongs.

• Knowledge of user context.

5.7.2 User context awareness User context awareness can be exploited to advantageously reduce the required level of explicit HCI. User context awareness and context adaptation can vary from passive to active modes. In passive mode, the system provides short lists of tasks and their user constraints that are relevant to the current situation. In active modes, the system performs the setting, e.g. B. Automates the rest of a task and reduces user involvement in completing the task. User contexts specify all physical, ICT and human environment context constraints related to the goal of a user task, e.g. B. to watch a movie. Contextual restrictions of the physical environment include the use of certain ambient lighting settings. The contextual constraints of the ICT environment include watching the movie on a large high definition screen with a surround sound system. The contextual constraints of the social environment dictate who the movie is watched with. A user context can contain the following properties:

• The physical characteristics and abilities of HCI users, eg. B. how easy it is for them to interact with a • • • • •.

some type of user interface, such as B. a pointing device. User presence in a location or some activity detected within the context of an application. User Identity (Section 12.3.4). Tasks and goals scheduled by the user (Section 5.6.4). Tasks related to user activity, which can be spontaneous and unplanned, can be concurrent, can involve compound tasks, and can be distributed across multiple devices (Sections 5.6.5, 5.6.6). The emotional state of the user (Section 5.7.5), p. B. repeatedly pressing a key can indicate impatience.

5.7.3 More Intuitive and Personalized Interaction MTOS-based devices tend to use a desktop UI metaphor along with using a file name as an index to organize information. To start working on documents, users need to remember how they categorized their documents in terms of file names and folder names and the devices on which they stored the files. This approach has several limitations. The information does not clearly fall into one category because the categories overlap and are confusing. It is impossible to generate categories that remain unique over time, and names are an inefficient way of categorizing information (Lansdale, 1988). Freeman and Gelernter (2007) propose to use virtuality based on the unification of visual expressions as a principle for better personal user interface design, instead of using physical metaphors as the basis for user interaction, since the latter can unnecessarily restrict design. Freeman and Gelernter identified the following principles for a better personal information system. Storage should be transparent and avoid the need for file and folder names. The file must be automatic. Reminders should be an integral part of the desktop. Personal data must be automatically available from anywhere and systems must provide useful summaries of documents. In the Lifestreams project, Freeman and Gelernter proposed the chronology, i. h Past, present and future as an organizational structure for information management. Freeman and Gelernter also differentiate between search and navigation. While powerful desktop search engines do exist, users need to be able to specify what to search for, but often they do.

iHCI Design

169

Don't do this, so it's best to support a powerful browser engine that supports user-directed navigation. Moran and Zhai (2007) present a more comprehensive analysis of the approaches studied to support more powerful personalized information. This is based on novel information organization principles associated with chronology, tasks, and multiple user-defined information associations. It is based on considering a more fluid interaction between individual and social interaction. It is based on the user's participation in multiple activities instead of just one. It is based on high-level activity and goal-oriented computing rather than lower-level task-based computing. Therefore, Moran and Zhai propose seven principles to turn the desktop information model into a very powerful model that can be used to support user-centric interaction in UbiCom applications: 1. From Office container to cloud of personal information: personal information, such as preferences, is not limited. to specific folders and files that contain the information within specific devices and services, but follow the user and can be used anywhere, anytime; see also virtual user environments (Section 4.2.1). Personal data must be personal, persistent, ubiquitous, secure and traceable, and in a standardized format. 2. From the desktop to a diverse set of visuals: A variety of visuals, adaptable to specific problem areas and different device form factors, are needed to complement the basic metaphor of the traditional desktop, since working on the desktop is just a part of UbiCom's reach. interaction. 3. From interacting with one device to interacting with many: To support the use of multiple instances and multiple types of devices, resources and services, design patterns that separate the view from the model are required. Designs are also needed to deal with the main inherent characteristics of multiplicity (Section 9.2). 4. From mouse and keyboard to more interactions and modalities: Developing more natural interaction with ICT devices is a key element of UbiCom's vision, represented by the support of iHCI ownership in UbiCom systems (Section 5.7). 5. Features can be moved from apps to services: Some desktop apps are clearly too complex for many users, while web apps, by contrast, are simpler, partly due to limited standardized interaction support on Web 1 Web 2 technologies enable richer web interaction. However, this may not necessarily work in volatile service environments (Section 3.3.3.9), and there is still the issue of multiplicity that must be addressed when binding interactions to services. 6. Personal to Interpersonal, Group to Social: Many personal interactions are often also part of multiple types of interpersonal interactions. Support is needed to make this more seamless through better social media support (Section 9.4.1) and by allowing more personal information to be shared socially, e.g. a mapping application where a user can share spatial annotations with others in a controlled manner (Liang et al., 2008). 7. From low-level tasks to high-level activities: User activities rarely have a clear beginning or end; Interruptions are to be expected; multiple activities running simultaneously; Contexts like time are useful for filtering and matching: however, contexts like time are rarely represented in computer interfaces; Associative information models are needed to support and link activities.

5.7.4 Personalization Personalization tailors Applications and Services specifically to an individual's needs, interests, and preferences. It may also involve personalizing a consumer product, electronic or written medium based on personal data or characteristics recorded by or on behalf of a user or consumer in the form of a user profile, e.g. a favorites list for viewing AV content. The profile does not necessarily have to be provided by the subject of the user, but rather is collected by observing a user's interactions without their knowledge. There are several prominent uses of personalization, including

170

Human interaction with the computer.

targeted marketing, product and service personalization, information filtering, and personalized customer relationship management (CRM). Personalization aims to develop a more complete user context model that is more reusable and persistent: across different user sessions or instances of the same type of user tasks, e.g. B. Back orders; via different user tasks, e.g. B. The user's personal data, such as their home address, can be reused between tasks; through different users, e.g. B. by allowing a user with little experience with a new task or with little knowledge of the context to use the experience of other users who are more familiar with a task or have more knowledge. Two key issues are, first, to design a personal preference model in such a way that it can be distributed and shared among multiple applications and users; second, to account for task-based and dynamic user preference contexts rather than using a static user context that acts as a lowest common denominator context applicable to multiple users across multiple tasks. One of the perceived benefits of customization is efficiency. Service interaction that can identify users and match previously collected user information, e.g. B. You can enable service invocation with one click. A system that makes it easy for the customer to take additional services and repeat the same service can potentially increase business volume. You can better serve customers by anticipating their needs. It can reduce user information overload (Maes, 1994). This is of particular importance in environments and interactions with limited resources. It can improve the usability of the services. You can make the interaction user-centric and more user-satisfying. It offers more suitable or interesting options for the client. It has a social dimension and treats the customer as a known person and not just another anonymous customer. Personalization can help users who have a partial view of the environment using access devices with limited screen and network capability from mobile devices such as mobile phones or handheld computers who have varying degrees of blindness or partial control of the environment. A possible solution for this, in addition to adapting to the current location and the type of access terminal, is the adaptation of the services and contents to the personal interests of the user. The adaptation of the services and contents to personal interests filters the available information. The filtering process is based on a user profile that describes that user's interests, skills, and characteristics. Personalization tailors content to specific user viewers. A flexible way to do this is to customize content properties based on viewer preferences, stored in a user profile. Potential downsides to personalization include loss of anonymity, loss of privacy, loss of user control, negative consumer discrimination, reduced choice, and increased information burden. There is a trade-off between the loss of privacy and the potential gains from personalization. If a provider's organization knows about the person, a service provider can resell this knowledge to other providers. A provider could manipulate the person by making recommendations to the detriment of the customer, e.g. B. providing attractive recommendations linked to easier loans that allow impulse purchases by customers who may have difficulty paying the amount borrowed. There is a loss of user control because the provider could create a model without the user's knowledge and the provider could create a model without the user's permission, so it is illegal. Finally, there is an opportunity for sellers to maximize their profit by making different offers to different types of customers, e.g. B. New customers are offered a cheaper offer for the same product than existing customers. Personalization tends to filter information and services to the user that are instances of or resemble personal preferences. This, by definition, prevents users from expanding their choice and experience of new things – personalization can keep us in our comfort zone. Personalization can cause more information load instead of less, because every time clients specifically call a service, e.g. B. the purchase of an article, several, in the worst case, non-discriminatory additional services are activated by suppliers that are perceived as appropriate to their interests. That can be distracting. Also, because providers store customers' personal information, they can offer their service remotely and often through multiple channels. Therefore, in practice, multiple levels of filtering are needed to filter options and filter unwanted offers.

iHCI Design

171

5.7.5 Affective Computing: Interactions Exploiting Users' Emotional Context An important human trait used in human-human interaction is the ability to recognize, interpret, process, and share human emotions. In 1995, Picard at MIT proposed the idea of affective computing, which relates to, arises from, or influences emotions. Affective computing applications have included computational learning, perceptual information retrieval, arts and entertainment, and human health and interaction. Emotional reactions make a central contribution to human behavior. The design challenges for affective computing overlap to some degree with the design problems of determining user context and those of developing more complex human-like intelligence models for use in UbiCom systems. Furthermore, human intelligence models (Russell & Norvig, 2003) are a central model for building artificial intelligence along with rational intelligence models. Picard reviewed some of the key design challenges for this paradigm and identified six design challenges (Picard, 2003):

• The range of means and modalities of expression of emotions is very wide: Many of these modalities •

•

they may be inaccessible (eg, blood chemistry, brain activity, neurotransmitters), and many others cannot be distinguished. The human expression of emotions is highly idiosyncratic and variable: it is challenging to accurately discern a person's emotional state from the available data. However, emotions can be classified more accurately if they are determined over time and correlated with other factors, such as the time of day. Cognitive models of human emotions are incomplete (little real progress has been made in cognitive modelling). Existing emotion models often use highly stylized stereotypes of personality types and emotional responsiveness that do not reflect the real-world behaviors of real people. The role of situational factors in emotional expression is poorly understood. The sine qua non for the expression of emotion is the physical body, but computers are not embodied in the same way. So, since computers are not embedded, they cannot reliably and believably express emotions. Computers can express emotions without having physical bodies, as seen in some characters in movies and animations. Also note that people with varying degrees of physical impairment can express emotions in a variety of ways, even with very limited modalities. Emotions are, ultimately, personal and private: they shed light on the most intimate motivations and responses. Attempts to read, detect and manipulate a user's emotions can invade the user's privacy. However, people often try to manipulate other people's emotions and this is not considered unethical. There is no need to contaminate purely logical computers with emotional responsiveness. However, several studies have supported the crucial role of emotions in many background processes: perception, decision making, creativity, empathic understanding, memory, as well as in social interaction (Picard, 2003).

5.7.6 Heuristics and Design Patterns User interface design should aim to support a conceptual system model based on HCI principles that supports good usability and conforms to a clear mental model of the user. There are many overlapping ad hoc sets of high-level HCI design heuristics proposed by many different HCI developers to promote good HCI interaction design. Specific guidance is required to apply these heuristics to design user interfaces that conform to these HCI principles. UI Design Patterns define a set of high-level design heuristics to support HCI usability principles and then map them into more concrete lower-level design patterns (Tidwell, 2005). Tidwell (2005) has identified a number of user subinterface design patterns that align with visual information accessed through desktop web browsers and mobile phones. Design patterns based on it but oriented more towards iHCI-UbiCom interaction are given in Table 5.1. These can be related to the higher level design heuristics given in Table 5.2.

Human interaction with the computer.

172

Table 5.1 User interface design heuristics for UbiCom based on the high-level heuristics proposed by Tidwell (2006) Heuristics

description

Design impact

safe scan

Allow users to explore or navigate without getting lost or getting into trouble

Provide an opportunity to undo something and try something different.

Befriedigend1

Look for good enough options, not the best option

Make descriptions and labels informative

changes in the middle

Users can change their mind about what has been done

Start, stop and restart processes easily; Remember previously entered information

deferred elections

Users may not want to make a decision now, may not have enough information, or may want to skip unnecessary questions.

Highlight, select a small set of required entries; Using good defaults allows users to revert back to deferred fields

incremental construction

Many things evolve, it takes experience to do something multiple times, it doesn't get done right the first time.

Make it easy to achieve your goals piece by piece; constantly give feedback to users

habituation or association

Users can make the wrong decisions as they get used to an input pattern that varies by app and device.

Standardize common gestures or interactions everywhere; Support to customize the interaction.

Constituted Shares

Lower level actions within a context count as other higher level actions, e.g. B. the use of keyboard shortcuts, multiple touches, gestures, etc.

Supports low-level definitions of action sets to give high-level meaning

proxy user interface

Users may need to use multiple individual devices and interfaces to access a composite service or UI

Support for the use of proxies to simplify access to multiple individual interfaces

Context-based reminder

Users manipulate objects frequently and according to context, e.g. B. when and where they used something, not by what it's called

Save context with objects. Use predictable contexts

Prospective Storage Contextual Triggers

People mark objects in their environment to remind themselves to take care of them later.

Supports proactive tags Supports proactive and situational task suggestions

help set up

Access to Help is adapted to different levels. The provider's help cannot be understood by the users. Help may not relate to user experience

Supports multilevel help. It allows the customization of the concepts of Help to the user. Help link to communities of practice

immediate feedback

Users can unnecessarily interrupt or reconfigure tasks or make redundant inputs when they cannot see the effect of their inputs.

Let users see the immediate impact of the actions they take and provide positive feedback; Avoid unnecessary steps

contextual explanations

It's often not clear why UI options aren't available or why actions don't work

Enable users to get in-context explanations of allowed or prohibited actions

Anticipated and Prospective Actions

Once a task sequence begins or continues beyond a certain point, it can become clear which tasks will follow to achieve a goal.

Supports proactive suggestion of automated tasks and partial completion of tasks by the system

iHCI Design

173

Table 5.1 (continued) Heuristics

description

Design impact

Optimized playback Optimized input

Facilitate repetition of user actions by detecting redundancies and optimizing playback

Just repeat. Recognition of repeating patterns and advice to users See techniques in Sections 5.2.3.1, 5.3

optimized output

Output may be limited due to lack of UI space or inability to access a specific UI model

See techniques in sections 5.2.3, 5.3 Show extras if necessary

Note: 1 Satisfy, derived from a combination of Satisfied and Enough, was proposed by the social scientist Herbert Simon in the 1950s. Given the uncertainty of the future and the cost of collecting information in the present, the extent to which agents can making a completely rational decision is limited. So the agents only have one

Table 5.2 Some examples of lower-level HCI design patterns linked to higher-level HCI design heuristics, based on Tidwell's (2005) set of patterns.

Low-level design pattern

high level heuristics

Organization of the set:

Wizard: Guides the user through the user interface in the prescribed order. Extras on demand: show important content, hide the rest. Multi-level help: use mixed help to meet different user needs

Help Minimalist Help

Cumin

Clear entry points: Present some descriptive entry points. Global Navigation – Place clear navigation links everywhere. Breadcrumbs – display a main and top page map on each page

Safe exploration Habituation Safe exploration Safe exploration

Arrange the design of a piece

Visual Framework – Design each page with the same basic layout

habituation

Decks of Cards - Use separate areas, stack them so only 1 is displayed. Responsive Activation - Start with a mostly off UI and activate more as the user progresses

minimalist minimalist edition

Smart Menu Items – Customize menu labels to reflect what they do when invoked. Cancel – User can cancel long tasks without any side effects. Progress Bar – Shows users the level of progress. Multilevel Undo – Provides a way to easily undo a series of actions performed by the user. Macros: users can create their own scripts

minimalist

do stuff

Receive user contributions

Forgiving Formatting – Allow the user to enter input in different formats, the app interprets them intelligently. Notice – Pre-populate a text box with a notice telling the user what to do

minimalist

(Video) Human Computer Interaction Class, Lecture 01 of 27

Error Handling Visible State Error Handling Optimized Retry Error Handling Help Minimalist

(continues overleaf)

Human interaction with the computer.

174

Table 5.2 (continued) Group of standards

Low-level design pattern

high level heuristics

Dropdown Selection – Expand the menu using a dropdown panel with a more complex value selection UI. Good Defaults: Fill out forms with the best guesses the user wants. Associated error messages: and if possible, place flags next to the original controls

Minimalistic and satisfying error handling

The user interface patterns listed in Table 5.2 are at a lower level than heuristics or user interface guidelines, but are still too high a level to be implemented via the API of a software toolkit. specific user interface to display display elements such as buttons and create a text box and set configure event handlers to support user interaction. A more understandable way to associate the UI objects defined in a UI toolkit with the HCI pattern is to use a back-end software design pattern such as MVC26 or Model, View, Controller (Krasner and Pope, 1988). (Figure 5.8) . This decouples the view (screen presentation) from the model (application object) via the controller (the UI's response to the UI). The controller can use a subscription pattern or a notification interaction pattern to connect to the view when the model changes. The design heuristics summarized in Table 5.2 can also be used to partially support implicit interaction designs. This is called partial support for iHCI because support for some implicit interaction features, such as users' emotional context, user understanding, and physical properties, have not yet been modeled in the form of design patterns. Activities across a variety of devices and apps are supported by design patterns: Safe Browsing,

Enter description here

Begin

Advanced search

Back-End-Objektmodell

interaction model

MVC Pattern Toolkit Control

query string

query

GetView input template

Front-end user interface (UI) presentation mockup desktop search app

query

UI Task Model Get Image Search

access

Clear Cue Points Pattern Button Button

Text's box

Defined in the GUI Toolkit API

GUI event handler

Figure 5.8 Relationship to HCI design heuristics

The MVC pattern is supported by many developer frameworks, such as Microsoft ASP.NET and WinForms.NET, as well as Java using the open source frameworks Struts and Spring.

references

175

Satisfaction, intermediate changes, deferred selection, incremental construction, habituation or fusion, constituted actions and user interface proxy. Contextual interaction is supported by: contextual memory, contextual stigmergy, situational help, contextual explanations, optimized repetition, prospective actions, optimized entry and optimized exit.

EXERCISES 1. Why is studying human-computer interaction important for ubiquitous computing? 2. Compare and contrast user interfaces and user interactions used with four common types of devices: personal computers, handheld mobile devices used for communication, game consoles, and remote-controlled AV displays. 3. Analyze designs to overcome the inherent limited input and output capabilities of handheld mobile communication devices. 4. Compare and contrast the use of explicit human interaction, implicit human interaction, and no human interaction for the following human activities: preparing a meal, driving a vehicle from start to finish, doing laundry, and washing your pet or yourself. 5. Describe an activity that you think cannot be automated digitally, then do a web search for ICT solutions that support it. 6. Discuss the disadvantages of a more natural interaction in a digital universe. Consider handwriting in terms of performance, accessibility, interoperability, etc. Then weigh the pros and cons of supporting mostly natural human-computer interfaces. 7. Using Fishkin's (2004) classification of physical user interfaces, classify the following types of user interface: computer mouse, any RFID-tagged object such as food, passports, etc., robots, and smart travel cards. 8. Discuss whether the proliferation of MEMS components could allow users to create their own custom tangible user interfaces for applications, shifting a current topic of computing research to mainstream consumer products. 9. Outline scenarios where specific user interfaces are beneficial and scenarios where their limitations outweigh their benefits.

Previous references, G.D. and Mynatt, E.D. (2000) Mapping past, present, and future research in ubiquitous computing. ACM Transactions on Computer-Human Interaction, 7(1):29-58. Bush, V. (1945) As we can think. The Atlantic Monthly, Vol. 176: 101-108. Reprinted and discussed in ACM Interactions, 3(2) (1996): 35-67. Bushnell, N. (1996) Relationships between entertainment and computer business. ACM Notices, 39(8): 31-37. Chapin, J.K., Moxon, K.A., Markowitzet, R.S., et al. (1999) Real-time control of a robotic arm with simultaneously registered neurons in the motor cortex. Nature Neuroscience, 2:664-670. Co, E. & Pashenkov, N. (2008) Emerging display technologies for organic user interfaces. ACM Notices, 51(6): 45-47. Dalisa, A. (1997) Electrophoretic display technology. IEEE Transactions in Electronic Devices, 24:827-834. Derrett, N. (2004) Heckel's Law: Insights Drawn from the User Interface Design of a Musical Device: The Bassoon. Personal and Pervasive Computing, 8:208–212. Dix A, Finlay J, Bowd G et al. (2004). Human-Computer Interaction, 3rd Edition Englewood Cliffs, NJ: Prentice Hall.

176

Human interaction with the computer.

Fishkin, K. P. (2004) A taxonomy and analysis of tangible interfaces. Personal and Ubiquitous Computing, 8(5): 347-358. Forster, W. (2005) The Encyclopedia of Slot Machines: Console, Laptop, and Home Computers 1972–2005. New York: schedule. Freeman, E. & Gelernter, D. (2007) Beyond Life Streams: The Inevitable Disappearance of the Desk Metaphor. In V Kaptelinin and M Czerwinski (eds) Beyond the desktop metaphor: Designing integrated digital work environments. Cambridge, MA: MIT Press, pp. 19-48. Goldstein, S.C., Campbell, J.D. and Mowry, T. C. (2005) Programmable matter. Computer Science, 38(6):99-101. Han, J. and Kamber M. (2006) Data Mining: Concepts and Techniques, 2nd Edition New York: Morgan Kaufmann Verlag. Harper R, Rodden T, Rogers Y, and Sellen A (eds) (2007) Being Human: Human-Computer Interaction by the Year 2020. White Paper, Microsoft Research Ltd. Available at http://research.microsoft.com/hci2020/ downloads/BeingHuman_A4.pdf, accessed March 2008. Harrison, B.L. (2000) Electronic books and the future of reading. Computer Graphics and Applications IEEE, 20(3):32-39. Hayles, N. K. (1999) How we became posthumans: virtual bodies in cybernetics, literature, and computing. Chicago: University of Chicago Press. Hofstede, G. (1997) Cultures and Organizations: Software of the Mind. New York: McGraw-Hill. Ihlström, C., A˚kesson, M. and Nordqvist, S. (2004) From Print to Web to E-Paper: the Challenge of Design the E-Newspaper. In Proceedings of ICCC 8th International Conference on Electronic Publishing (ELPUB 2004), Brasilia, pp. 249-260. Inbar, O., Ben-Asher, N., Porat, T., et al. (2008). All the right news for e-ink. Paper presented at the Conference on Human Factors in Computing Systems, CHI '08, Session on Research Landscapes, pp. 3621-3626. Inoue, S., Kawai, H., Kanbe, S., et al. (2002) High-resolution microencapsulated electrophoretic display (EPD) driven by four grayscale poly-Si TFTs. IEEE Transactions in Electronic Devices, 49(9): 1532-1539. IJsselsteijn, W. A. & Riva, G. (2003) Being there: the experience of presence in mediated environments. Emerging Communications, 5:3-16. Jacobson, J., Comiskey, B., Turner, C., et al. (1997) The last book. Systems Magazine, 36(3): 457-463. Jaimes, A. & Sebe, N. (2005) Multimodal human-computer interaction: a survey. In Proceedings of IEEE International Workshop on Human Computer Interaction in connection with ICCV 2005. In: Lecture Notes in Computer Science, 3766: 1-15. Johnston, RS and Willey, S. (1995) Development of a commercial virtual retinal display. In W. Stephens and LA Haworth (eds.) Proceedings of Helmet- and Head-Mounted Displays and Symbology Design, pp. 2-13. Jones, M. and Marsden, G. (2006) Mobile Interaction Design, Chichester: John Wiley & Sons, Ltd. Karray F and De Silva C (2004) Soft Computing and Intelligent Systems: Design: Theory, Tools and Applications. London: Pearson Books. Kim, D. and Kim, D. (2006) An intelligent control of the smart home through body gestures. In Proceedings of the 2006 International Conference on Hybrid Information Technology (ICHIT'06), 2: 439-446. Krasner, G. E. and Papa, S. T. (1988) A cookbook on using the model-view-controller user interface paradigm in Smalltalk-80. Object-Oriented Programming Magazine, 1(3):26–49. Kung, S.Y., Mak, M.W. and Lin S.H. (2004). Biometric authentication: a machine learning approach. Englewood Cliffs, NJ: Prentice Hall. Lansdale, MW (1988) The psychology of personal information management. Applied Ergonomics, 19(1):55–66. Lebedev, MA and Nicolelis, M.A.L. (2006) Brain-machine interfaces: past, present, and future. Trends in Neuroscience, 29:536-546. Lenoir, T. (2002) Makeover: Writing the Body in the Posthuman Technolandscape. Part One: Embracing the Posthuman: Configurations. Baltimore, MD: Johns Hopkins University Press and the Society for Literature and Science, 10:203-220. Liang, Z., Poslad, S., Meng, D. (2008) Adaptive and shareable custom geospatial mapping services for mobile users. Presentation at the GI-Days 2008 conference. Maes P. (1994) Agents that reduce work and information overload. ACM Notices, 37(7): 30-40. Mann, S. (1997) A historical account of the "WearComp" and "WearCam" inventions developed for "personal imaging" applications. In 1st International Symposium on Portable Computers, pp. 66-73. Mann, S. (1998) Humanistic intelligence: WearComp as a new framework for intelligent signal processing. In IEEE Proceedings 86(11), pp. 2123-2151.

references

177

Mann, S. and Fung J. (2002) EyeTap devices for intentionally increased, decreased, or otherwise altered visual perception of rigid flat patches of real scenes. Presence: Archives for Telemarketers and Virtual Environments, 11(2): 158–175. Marcus, A. and Gould, EW (2000) Crosscurrents: Cultural Dimensions and Global Web User Interface Design. ACM Interactions, 7(4):32-46. McTear M (2002) Spoken dialog technology: enabling the conversational user interface. ACM Computer Surveys, 34(1): 90-169. Mitra, S. & Acharya, T. (2007) Gesture recognition: a survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 37(3): 311-324. Moran, T.P. and Zhai, S. (2007) Beyond the desktop metaphor in seven dimensions. In V Kaptelinin and M Czerwinski (eds) Beyond the desktop metaphor: Designing integrated digital work environments. Cambridge, MA: MIT Press, pp. 335-354. Myers, B., Hollan, J. Cruz, I. et al. (1996) A brief history of human-computer interaction technology. ACM Computer Surveys, 28(4): 794-809. Navarro, K. F. (2004) Wireless and wearable brain computer interfaces in augmented reality environments. In Proceedings of the International Conference on Information Technology: Coding and Computing, ITCC 2004, 2: 643-647. Norman, D. A. (1988) The psychology of everyday things. New York: Essential Books. Orth, M., Post, R., and Cooper, E. (1998) Fabric Computing Interfaces. In Proceedings of the Conference on Human Factors in Computing Systems, CHI 98, Los Angeles, pp. 331-332. Picard, RW (2003) Affective computing: challenges. International Journal of Human-Computer Studies, 59:55-64. Pickering, J. A. (1986) Touch screens: Technologies and their applications. International Journal of Man-Machine Studies, 25: 249-269. Pignotti E., Edwards, P., and Grimnes, GA. (2004) Context-aware personalized service delivery. In European Conference on Artificial Intelligence, ECAI 2004, pp. 1077-1078. Preece, J., Rogers, Y. & Sharp, H. (2006) Interactive Design: Beyond Human-Computer Interaction 2nd Edition Chichester: John Wiley & Sons, Ltd. Pressman R.S. (1997) Software Engineering: A Practitioner's Approach, 4th Edition Maidenhead: McGraw-Hill. Raskin, R. (2000) The human interface. Read, MA: Addison-Wesley. Rekimoto, J. (2008) Organic interaction technologies: from stone to skin. ACM Notices, 51(6): 38-44. Rich, E. (1999) Users are individuals: individualization of user models. International Journal of Human-Computer Studies, 51:323-338. Riva, G. (2005) The psychology of ambient intelligence: activity, situation, and presence. In G Riva, F Vatalaro, F Davide and M Alcan˜iz (eds.) Ambient Intelligence. IOS Press. Available at http://www.ambientintelligence.org, accessed December 2005. Russell, S. and Norvig, P. (2003) Artificial Intelligence: A Modern Approach, 2nd Edition Englewood Cliffs, NJ: Prentice Hall. Schwab, I. and Pohl, W. (1999) Learning user profiles from positive examples. In Proceedings of the International Conference on Applications and Machine Learning, Chania, Greece, pp. 15-20. Schwesig, C. (2008) What makes an interface feel organic? ACM Notices, 51(6): 67-69. Sharma, R., Pavlovic, VI. and Huang, T. S. (1998) Towards a multimodal human-computer interface. IEEE Proceedings, 86(5): 853-869. Shneiderman, B. (1983) Direct manipulation: a step beyond programming languages. IEEE Computers, 16(8):57-69. Shneiderman, B. & Plaisant, C. (2004) User Interface Design: Strategies for Effective Human-Computer Interaction, 4th Edition, Read, MA: Pearson Addison-Wesley. Suchman L.A. (1987) Plans and situated actions: The problem of man-machine communication. Cambridge: Cambridge University Press. Sutherland, I. (1968) A three-dimensional head-mounted display. In Proceedings of Fall Joint Computer Conference, pp. 757-764. Thorp, E. O. (1998) The invention of the first portable computer. In 2nd International Symposium Wearable Computers: Digest of Papers, IEEE Computer Society, pp. 4–8.

178

Human interaction with the computer.

Tidwell J (2005) Interface design: patterns for effective interaction design. New York: O'Reilly. Van Laerhoven, K., Schmidt A., and Gellersen, H.-W. (2002) Context-aware clothing with multiple sensors. In Proceedings of the 6th International Symposium on Portable Computers (ISWC 2002), Seattle, IEEE Press, October. Vertegaal, R. & Poupyev, I. (2008) Organic User Interfaces. ACM Notices, 51(6): 26-30. Vidal, J. J. (1973) Annual Review of Biophysics and Bioengineering, 2:157-180. Warwick, K. (1999) Cybernetic Organisms: Our Future? IEEE Proceedings, 87(2): 387-389. Warwick, K., Gasson, M., Hutt, B.D., et al. (2003) The application of implant technology to cybernetic systems, Archives of Neurology, 60(10): 1369-1373. Wilson, A. and Shafer, S. (2003) XWand: user interface for smart spaces. In Proceedings of the SIGCHI Conference on Human Factors in Computer Systems, pp. 545-552. Yes, K-P. (2003) Peephole displays: stylus interaction on laptops with spatial awareness. In Proceedings of the SIGCHI Conference on Human Factors in Computer Systems, Ft. Lauderdale, Florida, pp. 1-8. Zimmerman, T. G. (1996) Personal Area Networks: Intrabody Near Field Communications. IBM systems. Magazine, 5 (3-4): 609-617.

6 Labeling, detection and control 6.1 Introduction As electronic components become smaller, faster and cheaper to manufacture, consume less energy and require less maintenance, they become easier to deploy widely and ubiquitously.1 Work in course on microelectromechanical systems (MEMS) 2 enable sensing and actuation down to the nanometer range. The possibilities of miniaturization extend to all areas of life, and the potential to incorporate information and communication technology literally everywhere is becoming a reality. Eventually, IT will become an invisible part of almost everything in everyone's environment, extending the Internet deep into the physical environment through embedded MEMS networks and making greater use of the expanded IPv6 address space. MEMS is a powerful enabler for the vision of intelligent ubiquitous computing environments. Although many mechanisms can be usefully reduced to integrate and fit into the physical world, some macromechanisms are still needed, often to support human activity, e.g. B. screens, passenger transport vehicles and many household appliances. Embedded systems are a second IT trend to enhance natural or man-made physical objects with networked digital embedded devices to provide better manual, semi-automatic and automatic detection and control, e.g. B. in motor vehicles. A third trend is the increasing annotation of the physical world, often driven by the need to enrich human and natural interactions in these environments in order to be better informed about them.

There are big social problems here (Chapter 13). As surveillance cameras get smaller, instead of being used to deter crime (crime is less likely when people know they are being watched), cameras can be used for casual surveillance. This could be used for economic ethnographic studies (Section 5.5.2), p. B. to observe what people look at in public space, such as in stores. It can also be used actively instead of passively to flag events that change the appearance of the physical environment to distract people when companies feel they are looking at things that have very little sales potential. 2 Note that while MEMS primarily refers to micro-sized mechanisms, in practice it often also includes millimeter- and nanometer-sized units.

Stefan Poslad

Mark, register and control

180

Nanotechnology Integrated Circuits MEMS Microactuators and Sensors

Robot-integrated RT process control

nanobots

Networked digital sensor networks

physical note

Integrated macro systems

RFID tracking devices

Figure 6.1 Enabling ubiquitous computing over embedded micro, macro, and annotation of physical objects in the world

Local and global interactions. For example, we might want to know where we left things like our keys or phones, or we might want to know where an insect we've never seen before came from. We often need to figure out where things are in the physical world in order to discover and trace their properties in order to virtualize views of the physical world and its physical objects. Virtual annotation model design can also learn from nature in this regard. Social insects, such as ants, leave chemical trails to support pheromone-based interactions with their environment and thus indirectly with each other (Holldobler and Wilson, 1990).

6.1.1 Chapter Overview This chapter describes three complementary development trends for generalized computing (Figure 6.1). First, tagging objects in the physical world using a combination of sensors and tags allows those features to be part of a discourse in a virtual computing environment. Second, the mechanisms can be reduced to millimeter, micrometer, and nanometer sizes for easier insertion into the physical world. The design aspects involved in creating and operating much smaller devices are considered from an ICT engineering point of view. Third, digitally activated and connected macromechanisms and devices will be accompanied by embedded process control computer systems, including robots, to automate human interaction and allow the systems to function autonomously.

6.2 Tagging the physical world Physical tags refer to digital tags, which are networked electronic devices with an identity, e.g. B. an RFID (radio frequency identifier) tag. When attached or linked to physical objects, they provide a way to examine physical spaces and processes. A key motivation for using physical world tags is to support context-based queries and tracking of physical world objects. Tagging is also an enabler for augmented reality (AR), which deals with the mapping of physical world objects onto computer artifact objects for more convenient access and manipulation in the virtual environment. AR environments can

Marking the physical world

181

be discussed from an HCI perspective (Section 5.4.2) or from an information perspective (Feiner et al., 1993). Here the focus is on the latter. Various techniques can be used to identify objects in the physical world and capture virtual views of the world. Digital identification tags can be added to objects, perhaps when creating artificial objects or even hatching natural objects. A virtual view of the world can be captured in the form of an image, video, or audio recording that is located in space and time, and features of interest in those views can then be identified and extracted. Specific physical phenomena in the world can be captured and recorded, e.g. B. a temperature change in space and/or time, and then linked to virtual views and associated data. Mobile capture devices such as phones and cameras are increasingly integrating location sensors to add location data to the time stamp for recordings.

6.2.1 Lifecycle for physical object markup Virtual markup comprises several separate sub-processes for data storage and creation, data processing, and data presentation:

• Capture a physical view or record physical objects or some characteristics of objects, such as: • moving or placing a reader3 within the range of a tag; • move tags within range of readers; • Capture a physical view of an object and its surroundings. This can also happen before

•

• • •

View cleanup processing to abstract the main features of interest from an object into a view, e.g. B. to isolate a voice in a noisy recording. Identify physical objects: • Recognize pre-assigned object IDs, eg. B. RFID, and look for what object has that identification; • Assign an ID to a physical view of the object, eg. image file ID, and then identifying the object within that view, perhaps by relative position, e.g. B. The object is the upper left rectangle of this image. Object anchoring or association: • Define the attributes and relationship of objects with respect to a physical view, such as an object. B. an object marked on a photograph of a part of the world; • Define the attributes and relationship of objects in relation to a virtual view, eg. B. an object marked in an abstract view as a spatial path. Organization or structuring: Objects associated with different objects within the same view, e.g. B. a map, and between different points of view they will be related. Representation: superimposition of pictorial annotations on physical views of the world, such as maps, or by different degrees of separation of annotations on different forms of the physical views. Management: Management of annotation data and processes, including creating, editing, deleting, recycling, storing, querying, and controlling access to annotation data.

6.2.2 Labels: types and properties There are many ways to characterize and classify labeling (Figure 6.2). Mackay (1998) describes three basic approaches to augmenting objects in the physical world for use in virtual (computing) environments: (1) augmenting the user by generating a digitized view of a part of the physical world while the user is in situ in that part is , p. use AR; (2) Enhance the physical object by attaching it directly or

A tag reader is a device that can receive and interpret RF signal transmissions from an electronic tag. If the tags are passive, the reader may also need to provide power for the tags to transmit their information.

Mark, register and control

182

objects in the physical environment

static states

dynamic states

AR sensor tags

Digital

analogous thing

expand users

virtual (note)

Physically

extend physical object

identifier

Physical-virtual label binding

Expand the physical environment.

On-site vs. off-site RFID

cardinality

Static vs Dynamic

Connected versus separate

adjunct

Separate

off site

virtual reality

context aware

In the place

RA, RFID

web resources

Figure 6.2 Taxonomy for tag types and properties

Embed a physical (digital) tag into the physical object, e.g. B. Use of RFID; and (3) enlarge the environment surrounding the user and the object, e.g. B. digitize and present a view of the physical environment in a virtual environment while the user is in that environment. Mackay made no distinction as to whether the virtual tag and the physical tag should be together on the site or not. Hansen's (2006) analysis focuses on the representation of the annotation. This also focuses on whether the user of the annotation is onsite (co-located or local) with the physical object or offsite (not co-located or remote). The virtual tag can be attached (or anchored) (connected to a physical tag) directly to the object to which it is related. Rather, the annotation may be separate (or not extended) or disconnected from the object to which it refers. Augmented Reality is seen as an example of users being on the site with a virtual tag attached. However, the uses of augmented reality can be broader: the focus is not on whether the user is in close proximity to a physical object with which they are interacting, but on whether the users' activities are located in the physical world or not instead of in a virtual world. VR is used as an example of an off-site user where annotations are attached to a particular representation of physical objects. Note that in VR the annotation does not refer to physical objects but to a virtual view of them. Virtual views can also be imaginary, not real, views of a physical environment, and may have no connection to any physical object. A web-based information system is an example of a type of tagging where virtual tags can be external and not anchored to a specific physical tag. In a context-aware application, the user is on the site, but receives an annotation that is associated with the physical object, but can be accessed remotely via a mobile wireless device (Section 6.2.5).

Marking the physical world

183

Hansen's classification of in situ versus attached labels complements Mackay's analysis. For example, Hansen's classification fails to distinguish the various ways in which the physical world can be expanded. In Hansen's analysis, user extension (eg, AR use) and object extension (eg, RFID use) are classified as in situ and anchored. Other topics include the cardinality of relationships between physical objects and their tags and notes on how physical tags can act as identifiers, as well as tag management issues. Physical tags can be used to identify, describe, and represent physical things in a virtual environment as virtual tags. Virtual tags are metadata, descriptions, or annotations about the physical object, which can then be manipulated in the virtual environment. These annotations are linked to the physical tag identifier. The annotation can be local to the physical tag in the physical world or external somewhere in the virtual environment, e.g. B. in a database are stored. Tags as physical identifiers could be associated with annotations and virtual tags that can also be identifiers, e.g. B. XML W3C Universal Resource Identifiers (URIs), which allow physical tags to be managed as virtual tags. Physical tags can be linked to multiple virtual tags. A physical tag can be linked to multiple virtual tags. However, not all virtual tags can be mapped to a physical tag. Physical tags are usually attached to physical objects that have a static state. If the state of physical objects changes, then it is better to also use sensors to determine the dynamic state of the physical object. The labels themselves must be recognized and identified. This is done with the help of a tag reader.

6.2.3 Physical and Virtual Label Management Enterprise physical object annotations still face many management challenges that make it difficult for mass consumers to use. Many of these problems are similar for sensors as well. Tags can be attached to objects when objects are created instead of adding them later. Tags can be permanently attached to objects instead of being removable. Tags could be removed from objects at the end of a usage lifecycle instead of never being removed. Tags can simply be discarded when the object is discarded (Section 12.4). Tags may need to be read outdoors in noisy, damp, dark, or very bright environments. Annotation data must be stored, distributed, and integrated based on the data collected; Data management should start as soon as the data is captured (via tag readers). There can be dozens of tags and readers per cubic reader. Readers can interfere with each other. Many redundant annotations of similar elements can be captured many times. One way to deal with redundant tag records is to use the event filter application to correlate events with business events, e.g. B. The package here means that the package was delivered to the local depot. Applications and businesses need to define the level of aggregation, reporting, and analysis. They may need to be dynamically reconfigured. Interoperability between different annotation systems may also be required. These design challenges are at the heart of ongoing research and development. The organization, structuring, and management of (annotation) metadata are discussed in more detail in section 12.2.9. Another dimension includes whether or not the physical object or, as the case may be, the human owner of the physical object is aware of the tag, whether or not the owner has sanctioned the use of the tag, i. h this raises privacy issues related to smart environments (Section 12.3.4).

6.2.4 RFID Tags Radio Frequency Identification (RFID) tags can be attached to objects to enable object identification around the world via a wireless connection. Unlike previous barcode technologies, it does not require line of sight or manual alignment to read the identification tag (Want, 2006). RFID tags are increasingly used to identify high-value goods. In 2009, however, barcodes are even more

Mark, register and control

184

more common than RFID tags for use with many low-cost retail items. Nath et al. (2006) review a series of static reader applications for automatically tagging, interrogating, and tracking rental cars, luggage, mail packages, and hospital patients. RFID tags can also be used in mobile readers embedded in phones to support faster contactless access to pay for local products and access local resources, such as hotel rooms. The main information that can be stored and retrieved in an RFID tag is the identifier data field itself4. B. Lot manufacturing information, including date of manufacture. RFID chips often work in a promiscuous mode: they respond to a generic scan instead of waiting for a reader to provide an activation code or other form of authentication. Many of the early adopters of RFID tags were retail companies, which used them in logistics operations such as Wal-Mart and Tesco. However, they have also been used by the military since World War II in the early 1940s when they were used by British military aircraft in IFF (Identification Friend or Foe) systems. RFID tags can be differentiated according to whether they have their own power supply (active RFID tags) or whether they are passive RFID tags that use the reader's power supply. Active tags are more expensive and require more maintenance, but they tend to have longer ranges compared to passive tags. A typical RFID system consists of two main components: the tag itself and a reader that scans the tag for its ID and contains additional computing, data storage, and communication facilities (Figure 6.3).

c) Figure 6.3 Application of RFID tags: (a) transponders in cars cause toll gates to open automatically when cars approach; (b) labels on pallets of merchandise tell traders where the merchandise is located; and (c) tags on clothing in retail stores can signal an alarm if removed without permission.

RFID tags used in the supply chain are encoded with an Electronic Product Code (EPC), which is a globally unique identifier for the object to be tagged. This is typically a 96-bit field, reserving some bits for a manager ID, some bits for the object type ID (24 bits, > 16 million object types), leaving 36 bits representing more than 68 billion of distinct instances of objects.

Marking the physical world

185

6.2.4.1 Active RFID Tags Active RFID tags are used on large items such as freight containers, rail cars, and large reusable containers that must be tracked over distances, such as a rail yard. They typically operate at 455 MHz, 2.45 GHz, or 5.8 GHz frequencies and typically have a 20-100m read range and cost $10-$50 (2005), depending on size of memory, the required battery life, whether that tag contains an integrated temperature sensor or other sensors, and the required robustness. There are two types of active tags: transponders and beacons. Active transponders wake up when they receive a signal from a reader. An important application of active transponders is toll collection and checkpoint control. When a car with an active transponder approaches a toll booth, a reader at the register emits a signal that activates the transponder on the car's windshield. The transponder then sends its unique ID to the reader. Transponders conserve battery life by allowing the tag to send its signal only when it is within range of a reader. Beacons are used in location-based systems (Section 7.4) where it is necessary to track the precise location of an asset within a region, such as a distribution station or along a transportation route. Longer-range location-based systems could use GPS or cell phone trilateration (Section 7.4). In a location-based system, a beacon emits a signal with its unique identifier at preset intervals (this can be every three seconds or once a day, depending on the importance of knowing the location of an asset at any given time). The beacon signal is picked up by at least three reader antennas located around the perimeter of the area where assets are being tracked. More complex active tags could also include sensors, e.g. For example, a tag could detect if a logistics item has been dropped or its ambient temperature has become too hot or too cold.

6.2.4.2 Passive RFID tags Passive RFID tags do not contain a power source or active transmitter, their power to transmit their information, typically between 10 mW and 1 mW, comes from the reader. They are cheaper than active tags, currently costing around 20-40 cents. They require less maintenance and have a much shorter range (read access) than active tags, typically a few cm to 10 m. A passive RFID transponder consists of a microchip attached to an antenna. Transponders can be packaged in many different ways, e.g. B. sandwiched between an adhesive layer and a paper tag to create a printable RFID tag; Embedded in a plastic card, key fob, and in special packaging that can withstand heat, cold, or harsh cleaning chemicals. Passive tags can operate at low frequencies (124 kHz, 125 kHz, or 135 kHz), high frequencies (13.56 MHz), and ultra-high frequencies (UHF: 860 MHz to 960 MHz). Low frequency tags are ideal for applications that require the tag to be read through certain soft materials and water at close range. As the frequency of radio waves increases, radio signals begin to behave more like light. They also cannot penetrate materials and tend to bounce off many objects. Waves in the UHF band are also absorbed by water. The big challenge for companies using UHF systems is being able to read RFID tags on boxes in the middle of a pallet or on metallic or submerged materials. There are two different approaches to transfer energy from the reader to the passive tags: near field and far field (Want, 2006). The great advantage of far field tags, as the name suggests, is that they can signal information at longer distances compared to the near field. Passive near-field RFID interaction is based on electromagnetic induction. An RFID reader passes a large alternating current through its electromagnetic coil (antenna), which generates an alternating magnetic field in its vicinity. When a tag with a smaller coil is placed in this field, an AC voltage will appear on the tag. This voltage can be rectified and coupled to a capacitor which can then

Mark, register and control

186

build up enough charge to power the tag chip. Similarly, the tag reader can use this energy to vary the magnetic field across its antenna to send a signal containing the tag's identification to the reader's antenna. Far-field passive RFID interaction is based on the detection of EM waves propagating from a dipole antenna attached to the reader. A smaller dipole antenna on the tag receives this power as an alternating voltage difference, and can in turn be charged with power. However, near-field magnetic induction cannot reverse the process of transmitting a signal from the tag to the reader because the field decreases inversely with respect to the cube of the distance between them, so backscattering is used instead ( Want, 2006).

6.2.5 Personalized and Social Tags The main physical world annotation examples considered so far, RFID tags, are targeted at business and organizational users. Already in 1945, before the advent of the era of digital computing, Bush (1945) raised in his Memex system the idea of personal notes and their use as part of electronic personal information as "a device in which a person stores all of his books, records and communications, and that it is mechanized so that it can be consulted with extraordinary speed and flexibility. It is an extended intimate addition to his memory.” What he did not explain was how the information was collected and used for the annotations. There are several current initiatives to comment on personal views of the physical world, eg B. Gemmell et al (2006) discuss a project called MyLifeBits to record all personal experiences of an individual Other initiatives such as Semacode (2005) propose a scheme to define labels that can be automatically processed from captured images and linked to a web-based encyclopedia of spatial information. Basically, a semacode encodes URLs as part of a 100-character string encoded in 2D barcodes. To create semacodes, a URL is entered and the software converts it into a semacode image that can be printed and attached to physical objects (Figure 6.4). Some management may be required to control the removal, movement, and attachment of malicious tags. Mobile devices may contain a Semacode scanner, which consists of a camera to photograph the code and software to read these codes, analyze them and download the associated resource to a user device. There are several key usage differences between business use of annotations and personal or social use. Physical artifact annotation is often driven by business objectives, ie h Reduce costs by detecting and recovering lost assets. Company annotations often uniquely identify objects, mark specific types of artifacts as they are produced, and use very simple alphanumeric codes to represent the artifact. The personal use notation is less specific, deterministic and

Connect to the physical world

Convert URL to visual code

photo (read code)

to receive

Website

post phone Figure 6.4 Augmented reality labeling processes

sensors and sensor networks

187

quite subjective, multimodal (over several sensory channels) and presented in multimedia. Subjective annotations are used in multiple contexts, multiple applications, and multiple user activities. A major challenge here is the so-called semantic gap between low-level object features that can be automatically extracted from a record of physical objects in their environment and their high-level meaning in relation to a context of use.

6.2.6 Micro and macro tags Most tags in the physical world, e.g. B. RFID tags are macro-sized tags that match macro-sized objects. RFID tags have the potential to be shrunk to hundredths of a millimeter based on MEMS techniques, which are much cheaper to mass-produce but invisible to the naked eye. The size of a radio antenna transceiver for the microtag that can be located outside the tag depends on the wavelength of the radio signal transceiver and is typically on the order of about 5 cm for a 2.45 GHz signal. MEMs and nanotechnology markers, we have the potential to mark the physical world and to observe and affect the world in unprecedented ways at very fine granular levels at the micro and nano or molecular levels.

6.3 Sensors and sensor networks Sensors are a type of converter that converts physical phenomena such as heat, light, or sound into electrical signals. Sensors often act as enablers, inputs to a system's behavior to allow it to adapt more favorably, often embedded as part of a control loop into pre-programmed systems that perform specialized rather than general functions, such as: B A temperature sensor can be connected to a heating system. Sensors can be used to: instrument and monitor environments; asset tracking through time and space related to a workflow or process; recognize changes in the environment that are defined as significant and that people cannot perceive or are at risk of being perceived; Control of a system in relation to the environment within a defined range of change; Personalize the Services to enhance their usefulness. Sensors like RFID tags are networked. The basic architecture is similar: sensors can act as data generators, intermediate or serving nodes can receive, process and store data, possibly remotely. However, while the tags only produce a fixed electrical signal, the sensor data can change as it is the output of a transducer that converts different physical phenomena into different signals. Data processing is also likely to be more complex with sensors than tags, see below. Sensors can range in scale from nanosensors to macrosensors, such as B. A windsock used to indicate wind direction.

6.3.1 Overview of sensor network components and processes The main components of a typical sensor network system, shown in Figure 6.5, are sensors connected in a network served by a sensor access node. 5 A slightly different but compatible view of a sensor network is composed by considering sensors as three types of nodes: (1) common nodes

5 The concepts of sensor node and sensor network can be somewhat ambiguous. A sensor can act as a node in a sensor network (Akyildiz et al., 2002) as opposed to a dedicated sensor network server receiving data from multiple sensors, often referred to as a sensor (access) node (Zhao and Guibas , 2005). There are sensor networks that are connected to and served by a single server node, and sensor networks that consist of multiple servers or access nodes and sensor networks, i. h networks of networks.

Mark, register and control

188

physical phenomena p

Range of distribution of the phenomena that can be measured

S S

storage

sensor network

S S

Sensors that detect events

Username

access node

Sensors that notify the internet of the access node

Figure 6.5 A sensor network used to detect heat increases and report them to a user

Mainly responsible for collecting data from sensors; (2) receiving nodes, which are responsible for receiving, storing, processing, and collecting data from common nodes; and (3) gateway nodes that connect receiving nodes to external entities. Common nodes are equivalent to sensors, and access nodes combine the functionality of the receiver and gateway nodes. In addition to the access node, some sensors in the network can act as sink nodes within the network. Section 6.3.5 considers the benefits of more distributed data storage, retrieval, and processing. Sensors are transducers that convert a physical phenomenon such as heat, light, or sound into an electrical signal. Energy is a key concern, and communication is the main consumer of scarce energy resources, more so than processing. Sensors often have a short-range, low-power wireless interface that allows them to communicate with other sensors within their range and with receivers or data readers, also called sensor nodes. In Figure 6.5, the sensors could work together such that only a single sensor source and the sensors send data along a single path to conserve power. Sensor access nodes multiplex data from multiple sensors and often also support a local controller with a microprocessor and signal processor. Sensor nodes can also support local data logging and storage. The size of the sensors varies from micro to macro size. The size of the sensor nodes range from the size of a shoebox to the size of a pencil box to the size of a matchbox. The sensor access node acts as a "base station" that sends requests to other appropriate sensor nodes in a sensor network. In the illustrative example in Figure 6.5, three sensors are in range of an event, two sensors are damaged by the event, and two sensors are in range of the access point. Because the input event sensors are not in range of the access node, they must route their data through other sensors to reach the access node in order to access the events over the Internet. Sensor networks can contain a large number of sensors and nodes. Sensor networks can be heterogeneous in terms of sensor node types and sensor types; This makes interoperability a key concern. Managing all of these constraints and creating a system that works well for the application domain, and at the same time understandable and manageable for human operators and users who may be casual passers-by is a huge challenge. Estrin et al. (2002) characterize sensor design problems with respect to spatial and temporal scale, variability, degree of autonomy, functionality, etc.

sensors and sensor networks

189

Complexity. This characterization has been extended and the autonomy property of the sensor has been merged with the functional complexity. Sensors can be characterized in terms of:

• Characteristics of the perceived phenomenon: The type of physical phenomenon perceived includes

• •

•

location, temperature, etc. The user context defines the threshold, range, history, e.g. For example, when a resource is accessed more than a certain number of times or when a threshold is exceeded. The spatio-temporal distribution in the environment can be defined. If a single sample is measured, this may not be enough because some phenomena vary in space and time, that is, they move. Thus, this affects the sampling interval, the extent of overall system coverage, and the relative number of sensor nodes for the input stimuli. variability: the type of environment detected; application tasks; the spatial distribution, i. h Interesting objects in the environment that can be moved. Physical characteristics of the sensor, including performance, mobility, and size. Passive sensors transmit power from a reader, while active sensors require their own power source. Sensors can be fixed in place or mobile. The sensors may be anchored and move with some part of the environment, such as B. an animal, or the sensors may be untethered, i. h in the air or in the water. Sensors can range in size from nanometers up. Functional complexity: some sensors do not have autonomy and have simple functionality. Sensors simply convert some physical phenomenon into data that is easily reported to human users. Sensors can have more autonomy and be pre-configured to automatically detect pre-defined events. Sensors can be reconfigurable, self-configuring, and self-optimizing. Multiple sensors can work together on site, e.g. For example, more than one acoustic sensor is often used in different acoustic systems, one to detect background noise and one to detect a signal, and combining data from both can help improve the signal-to-noise ratio. Sensors can also be used as part of an integrated control system.

The challenges in the design and implementation of sensors and solutions are summarized in Table 6.1. The main functions of sensor networks can be superimposed on a protocol stack according to the physical properties of the network, data network properties, data processing, and sensor choreography (Figure 6.6). Each of these will be discussed in turn. Instead, other conceptual protocol layer stacks could also be used to model the operation of the sensor, e.g. B. Horizontal layers for physical, data link, network, transport, layer, and application along with some generic vertical layers for power management, mobility management, and task management, each interlocking with the horizontal layers (Akyildiz et al., 2002). ). However, the protocol stack in Figure 6.6 attempts to emphasize the need to flatten the horizontal protocol stack to support fewer errors in data transmission and to consolidate managerial control over sensor data collection and processing. Note that this is not each of these components, e.g. B. data processing, restricted to distribution.

6.3.2 Sensor Electronics Figure 6.7 shows a block diagram of a circuit for a sensor. It is divided into four main functions: Acquire, Process, Send and Receive, and Performance Related. The sensor signal is filtered and amplified, converted to a digital signal by the analog-to-digital converter (ADC), simple digital signal processing (DSP) is performed on the sensor before the signal is modulated for transmission. This special sensor design also supports input configuration for the DSP. The MEMS sensor design can reduce the size and power consumption of the sensor by combining several separate electronic components on a single chip. Sensors often need to be able to run low-duty, long-lived, unattended systems.

Mark, register and control

190

Table 6.1 Sensor design and implementation challenges and some corresponding solutions Challenges of a sensor network system

design solutions

Sensor power is a scarce resource for data transmission

Use a sensor network that uses short distance, low power transmissions. Connect sensors in mesh networks and use multi-hop transmissions. Filter data on site and transmit only filtered data Sensor density around estimated signal source locations when deterministic; Make sensor layouts reconfigurable, self-organizing, and mobile. It supports variable sampling and supports burst data acquisition. Use dense, low-power sensor networks with redundant paths to route data across the network to locate sensors (in motion) and can be used to place them. Use specialized routing protocols to control dynamic mesh topologies to work with.

Limited capacity for storage and computation in sensors Dynamic and non-deterministic spatial-temporal distribution of events. You may not be able to determine in advance the best way to use the individual sensors

Sensor failures often occur due to lack of power, physical damage, active (noise) or passive environmental interference from the transceiver, access node, or non-optimal positioning. Multihop sensor networks can have a dynamic topology. There may not be a global understanding of the network structure. Sensors can be too expensive to upgrade once deployed.

Design sensors and sensor access nodes to be low maintenance. Support sensor redundancy Use on-premises computing at both the sensor and sensor access node

Sensors can generate huge amounts of data.

Definition and processing of events

collaborative editing

data discovery

data processing data storage networking

sensor a red

electronic sensor

on-site processing

Intra Routing vs Intranode Sensor Distribution and Density DSP

data insecurity

addressing

HF, optical transmission properties

Characteristics of the physical environment

power management

Figure 6.6 The most important functional characteristics for the use of sensor networks

sensors and sensor networks

191

Process Storage Sensor Converter

analog filter amplifier

ADC

DSP

B. Thresholding, FIR/IIR filtering, statistical analysis, FFT transceiver antenna

Modulator Power Power Management

demodulator

Sender

Change

container

Drums

Figure 6.7 Block diagram of the electronic circuit of a sensor

6.3.3 Physical Network: Environment, density, and transmission sensors can be deployed in three phases: (1) a pre-deployment phase that involves breaking up groups of multiple release points and dispersing or individual placement; (2) a deployment phase during which sensors can be moved after deployment due to changes in signaling phenomena, power optimization, or task changes; and (3) a redeployment phase in which additional sensors are used. There are several reasons that motivate the use of multiple low-cost, short-range, and low-power sensors, instead of using a few long-range, high-performance, and high-cost sensors. First, dense sensor networks can improve signal-to-noise ratio (SNR) by reducing the average distance of the sensor from signal sources of interest. Each sensor has a finite detection range, which is determined by the reference noise level (ground). A dense sensor field improves the probability of detecting a signal source within range. Once a signal source is within the detection zone, increasing sensor density further reduces the average distance between sensor and signal source, improving SNR. As an example, consider acoustic perception on a two-dimensional plane (Zhao & Guibas, 2004, pp. 6–9). The received sound power Pr at a distance d from a sound source Ps varies inversely with the square of the distance between them (equation 6.1): Pr / Ps=d2

ð6:1Þ

The SNR of the received power signal at the noise level signal power level Pn is expressed on a logarithmic scale of decibels: SNR ¼ 10 log ðPr=PnÞ ¼ 10 log Ps 10 log Pn 20 log d

ð6:2Þ

Increasing the sensor density by a factor k inversely reduces the average range to target by the square root of r. Therefore, the SNR benefit of a denser network is given in Equation 6.3. pﬃﬃﬃ pﬃﬃﬃ SNRdif ¼ SNRðd= kÞ SNRðdÞ ¼ 20 logðd=ðd= kÞ ¼ 10 logk ð6:3Þ Increasing the sensor density by the factor k improves the SNR on a sensor by 10 log dB for acoustic signals. Sensors must be distributed so that they can maximize coverage areas and be local enough to detect strong enough signals of the physical phenomenon of interest. They should be arranged in such a way that adjacent sensors overlap a bit

Mark, register and control

192

Network coverage so they can find a data transmission path to a sensor node. Sensor distribution and coverage should be arranged to match the distribution and coverage of the physical phenomena of interest to optimize detection. However, using a uniform sensor distribution may not always be optimal due to, for example, uneven signal attenuation and obstacles. Increasing the sensor density may have no effect if the density of the obstacles is similar to the density of the sensors. It may not always be possible to determine in advance an optimal distribution of sensor deployment. Therefore, there is a degree of perceptual uncertainty (Kaiser et al., 2005). A random bet could also be used, e.g. B. in inaccessible terrain or in disaster operations. Additionally, mobile sensors can self-organize into an optimal configuration. Second, the energy efficiency for communication can be increased by using a multi-hop topology for the sensor network (Zhao and Guibas, 2004). In a network of N hops, the total distance for transmission is Nd, where d is the (average) distance of one hop. The minimum received power is Pr and the power at the transmitting node is Ps; a is the RF attenuation coefficient, typically in the range 2 to 5 due to multipath and other interference effects: Pr / Ps=da a

$/day Pr

ð6:4Þ ð6:5Þ

Therefore, the performance advantage of an N-hop transmission over a single-hop transmission over the same distance Nr, Pdif, is given by equation 6.6: Pdif ¼ PsðNdÞ=ðN : PsðNdÞÞ ¼ ðNdÞa Pr=ðN : da PrÞ ¼ Nða 1 Þ

ð6:6Þ

However, the increase in performance must be weighed against the disadvantages of using more relay nodes: the higher power consumption of all components, the higher cost of using more sensors, and the higher latency of relaying messages over multiple jumps. In addition, power management can also be assisted by optimizing routing or management processing to account for transceiver shutdown on redundant sensors and sensor paths. Relevant information can be added during multi-hop data exchange, and when sensors support multi-path data exchange, the system is reasonably resilient to single sensor node failures. Akyildiz et al. (2002) consider network access protocols that include signal modulation and demodulation in more detail.

6.3.4 Data Network: Addressing and routing nodes in any type of distributed system may need to be uniquely addressable. In most distributed systems, the address of the nodes uses the topological location of a node in the network. The (virtual) network topology can also differ from the physical topology. In IP-type fixed networks, the addressing scheme includes IP assignment and hierarchical hostname lookup, multicast packet data interleaving, and routing with service registration and lookup. Node discovery uses an attribute-based addressing scheme that is independent of the network topology. Typically, network nodes are addressed in terms of the properties of the resource at that node, which are defined by a set of attributes, e.g. B. those on the “southernmost” edge of the covered region. Service-oriented networks typically use multiple levels of indirection to address nodes. A key design issue here is how to efficiently design the node attribute to resolve the physical node address to support the low power requirements of massively distributed real-time sensor networks. An efficient attribute-based node addressing scheme has been proposed by Heidemann et al. He suggested. (2001) called, among other things, directed diffusion. This supports processing on the network to take advantage of CPU communication offsets for sensor networks, reducing the amount of bypass and working directly on top of low-level (hop-by-bop) communication protocols.

sensors and sensor networks

193

The basic idea of directed diffusion, i. h Data-centric routing consists of naming data (not nodes) with relevant external attributes, such as data type, time, node location, and SNR. This can support aggregation and processing of data sources on the network, publication of data by sensor sources, and subscription of data by data client nodes. One node can perform multiple roles, e.g. B. aggregate, combine and process data from incoming sensor nodes and become a source of new data itself. A node can act as a client to fire event data and then as a server that publishes data only when a combination of conditions occurs. Other directed data fusion routing protocols include the following. SMEC, which creates a subgraph of the sensor network that contains the minimum energy path. Flooding6 sends data to all neighboring nodes, regardless of whether they have received it before or not. By clapping, data is sent to a randomly selected neighbor. SPIN sends data to sensor nodes only when they are interested. SAR creates multiple trees, with the root of each tree being a hop neighbor from the access node. LEACH groups to minimize power dissipation. Sensor network routing algorithms can also be classified by type of network structure, such as flat or hierarchical, or by interaction protocol, such as multipath, query-based, or negotiation-based (Al-Karaki and Kamal, 2004).

6.3.4.1 Sensor networks versus ad hoc networks There are some differences between sensor networks and ad hoc networks (Akyildiz et al., 2002). Unlike ad hoc networks, sensor networks are more dense, containing orders of magnitude more nodes (sensors), which may not be uniquely addressable at the application level due to the large number of node instances. the same type. The topology of sensor networks can be very dynamic. The topology may need to accommodate locally detected sensors, unknown signal properties, (auto)reconfiguration of nomadic sensors in situ, and dynamic routing to optimize low SNR and low power consumption (Section 6.3.2). Sensor nodes typically use a broadcast communication paradigm, while ad hoc networks typically use point-to-point communication. Sensors typically use a mesh topology network (Section 11.7.8.5) that is dynamic, mobile, and unreliable, requiring neither universal routing protocols nor a central registry of sensor locations and routes. Each sensor acts as a router and data source and, depending on the version, also as a data sink and gateway.

6.3.5 Data Processing: Distributed Data Storage and Data Retrieval Sensors as data sources are naturally distributed and therefore computation is distributed as it is done locally to reduce it. Processing involves collecting events from massively distributed sensors. It would be easier for each sensor to act in isolation to store all data events, but this requires a large amount of storage space. A traditional centralized data warehouse approach could be used with many sensors generating data transactions that cause updates to a data warehouse. The data events extracted from the sensors could be stored in an RDBMS server. Query processing occurs only on the data server nodes. In addition, various RDBMS data could be aggregated into a data warehouse and SQL could be used to query data from sensors identified using an attribute-based addressing and routing scheme7.

6 high tide means diffusion. However, in the case of nodes with low resources (bandwidth and processing), streaming can easily overwhelm the capabilities of the nodes. Also note that streaming is used in other dynamic networks, such as p2p networks and ad hoc networks, to locate resources, peers, services, users, etc., as there may not be any fixed middleware services. 7 Traditional addressing and routing, address-centric routing, involves passing data from a source to its destination without interpreting the content of the data. By contrast, attribute-based addressing and routing evaluates the content of data transmitted at each hop in the network to determine how the data is routed.

194

Mark, register and control

However, many events may contain little valuable information because the readings are constant. Furthermore, multiple termination events can be generated to characterize a signal. Another alternative to storing each event is to filter the data events and transmit only the filtered events. This requires data filtering on each sensor. Furthermore, data processing, such as data aggregation, can more conveniently be performed in the network at or near the data source sensor nodes (Bonnet et al., 2000). This is exploited to reduce communication in data sink type nodes. It depends on the design if each shared sensor node behaves as a sink node or not. Sensor database systems should support distributed query processing in a sensor network and consider how the sensor database design can represent sensor data, represent sensor queries, process query fragments in nodes sensors and distribute query fragments. Theoretically, distributed database techniques could be used here. Sensor nodes can be clustered autonomously using attribute-based sensor addressing to support energy-efficient and scalable data transmission. Data aggregation processes could also be applied recursively to form a hierarchy of groups, e.g. B. the SINA (Sensor Information Networking Architecture) system of Shen et al. (2001). In SINA, Sensor Query and Tasking Language, SQTL, which can be adapted to Structured Query Language (SQL), acts as an API between sensor applications and SINA middleware and supports three types of events: (1) Events that occur when a message generated by a sensor node; (2) events triggered periodically by a timer; and (3) events caused by the expiration of a timer. A simple user query is: "SELECT avg(getTemperature()) AS avgTemperature". Another data storage system for sensor networks is Cougar (Bonnet et al., 2000). Application tasks may require node collaboration, as an isolated event may not be significant or may be too easily identified as a false positive or false negative. Node collaboration can also be used to filter duplicate events and disable active sensors that generate or transmit redundant or insignificant information. Communicating insignificant data at the application layer also wastes the scarce available energy and bandwidth resources. Therefore, sensor data access models could use utility functions that trade off the gain of one or more sensor nodes that generate and transmit data against constraints against the cost of power and bandwidth usage. in doing so. This is the idea behind the information-based sensor task (Zhao and Guibas, 2004).

6.4 Microactuation and sensing: MEMS MEMS (microelectromechanical systems) are electronic devices in the micron to millimeter range that are manufactured as discrete devices or in large assemblies (Berlin and Gabriel, 1997). MEMS perform two basic types of functions by acting as sensors or actuators. Both actuators and sensors act as transducers, converting one signal to another. Of particular interest are the transducers that convert some environmental phenomena such as temperature, humidity, pressure, etc. into a digital electrical signal and vice versa. MEMS actuators convert an electrical signal into physical phenomena to move or control mechanisms such as motors, pneumatic actuators, hydraulic pistons, and relays. MEMS sensors work in reverse to actuators, converting environmental phenomena such as temperature, humidity, and pressure into an electrical signal. These relatively small components have high resonant frequencies which result in higher operating frequencies. Aggregates of millions of cooperating mechanisms of perception, actuation, and locomotion can be seen as a form of programmable matter because they can self-assemble into arbitrary three-dimensional shapes (Goldstein et al., 2005) (Figure 6.8) that lay the foundation for a much more fluid form. and flexible computers and human-computer interfaces, p. B. Flexible devices and physical computer user interfaces (Section 5.3.4). A long-term goal of using such collections is to be able to achieve a synthetic reality that, unlike virtual reality and augmented reality, allows for the physical realization of all computer generated objects. This has the advantage that users can experience synthetic reality without sensory augmentations, such as head-mounted displays, and can physically interact with any object in the system naturally (Goldstein et al., 2005).

Microactuation and detection: MEMS

195

Figure 6.8 Some examples of MEMS devices, size on the order of 10 to 100 microns (from left to right): mite approach gear chain, polysilicon mirror, three-piston microsteam motor. Reproduced with permission from Ó Sandia National Laboratories, SUMMIT(TM) Technologies, www.mems.Sandia.gov

6.4.1 Fabrication The design of MEMS differs from that of equivalent macrodevices, which include the design of discrete mechanical and electronic components, as these microcomponents are based on the design of a silicon-based integrated circuit (IC), also known as a dad. Analog devices can also be replaced by IC versions, e.g. B. While a traditional thermometer is based on a liquid such as mercury expanding through a tube referenced to a calibrated scale, an electronic thermometer can be built from a thermocouple and amplifier IC. Integrated circuits consist of multiple layers of p-type and n-type doped silicon that are added to a substrate. An optical microfabrication approach, photolithography, is then used to fabricate the circuit. This first coats a layer of a light-resistant chemical. The circuit pattern to be produced is then drawn on a photomask. The photolithography system shines ultraviolet light through the photomask and casts a shadow onto a layer, which then reacts with the lightfast chemical and cures, allowing selective removal of portions of the substrate to be etched chemically. Therefore, engineers design a new circuit by designing the connection pattern between millions of relatively simple and identical components. It is the variety and complexity of the connections between these that creates the attendant variety of electronic components, including memory chips and CPUs. The miniaturization of IC-based MEMS processing has two important advantages over macroelectromechanical devices and systems: mass production and energy reduction. The multiplicity makes it possible to manufacture ten thousand or a million MEMS components as easily and quickly as one; these economies of scale are critical to lower unit costs.8 Second, the performance of integrated circuits improves when components are closer together as they switch faster and faster. you can use less power. The second part of MEMS that complements the microelectronics part is the micromachine part. Interestingly, these micromachines can be made as integrated circuits. MEMS-type ICs can be fabricated in a number of ways by: bulk micromachining (etching into the substrate); superficial micromachining (accumulation of layers on the substrate and engraving); and editing deep LIGA9 structures.

6.4.2 Microactuators The mechanisms involved in microactuation, while conceptually similar to the equivalent macromechanisms, can function in fundamentally different ways. They are built fundamentally

Gershenfeld (1999) even suggested the idea of proposing MEMS fabrication not in specialized facilities but through stand-alone desktop printers distributed among product developers. Instead of printers that deposit ink from ink cartridges, these printers deposit cartridge materials to form MEMS devices. He calls this Printed EMS or PEMS. 9 LIGA is the German acronym for X-ray Lithography, Electrode Positioning and Casting.

196

Mark, register and control

otherwise, using integrated circuit design and nanotechnology. Applications for MEMS actuators include:

• Micromirror array-based projectors (microprojectors) can be used to create a large screen

• •

•

View content from smaller devices. This finds application in navigation systems (Heads Up Displays or HUDs), as a positioning aid in medical diagnosis and treatment, and in manufacturing to establish reference points for drilling. A microsensor system may also be required to complement the use of microactuators, e.g. B. to detect and compensate for motion noise, such as camera shake, to ensure a stable image even when you're moving. Inkjet Printer Heads: MEMS can be used to control ink deposits on paper. Fuller et al. (2002) also suggest that MEMS devices can be printed and distributed on paper using an inkjet printer. Optical Switches: Optical Cross Connect (OXC) switches are devices used by telecommunications providers to switch high-speed optical signals in a fiber optic network. OXCs often have electronic cores and can become a communication bottleneck with increasing data rates, prompting the development of an all-optical MEMS switch (Yeow et al., 2001). Micro Fluid Pumps – Key components include a fluid actuator, fluid controller, and micro fixtures e.g. B. for use in drug delivery. One of the main challenges here is the selection of materials that can be used to fabricate biocompatible ICs, e.g. B. Parylene (Meng and Tai, 2003). Miniature RF Transceivers – can replace low-Q passive components in communication devices such as resonators, switches, capacitors, and vibration inductors, where the Q factor indicates the rate of power dissipation relative to the oscillation frequency, and set them to a just high. Q MEMS - HF transceiver chip. This allows for further miniaturization of the communicators, see Mansour et al. (2003) and Rebeiz (2003). Miniature storage devices – can support gigabytes of non-volatile data storage on a single IC chip, low power consumption and low data latency, e.g. B. Worst-case turnaround latency 5-11 ms, average access time less than one millisecond. Yu et al. (2007) consider how to optimize RDB storage in MEMS storage devices.

6.4.3 Microsensors Kahn et al. (2000) consider downsizing a priority to implement sensor nodes as cheaply and easily as possible, e.g. B. to integrate the required sensor, communication and computing hardware together with a power supply in a volume of not more than one cubic millimeter and still achieve reasonable performance in terms of sensor functionality and communication capability. These millimeter-sized knots are called "smart dust." Smart Dust can be small enough to float in the air, circulate air currents, and sense and communicate for hours or even days. The smart dust motes contain microsensors, an optical receiver, passive and active optical transmitters, signal processing and control circuitry, and a thick-film battery power supply. A fundamental part of the design is highly efficient power management in terms of energy storage and power consumption for both data acquisition, processing and transmission. The stored energy is about 1J, which is designed to be consumed at 10mW throughout the day. Performance-optimized CPUs typically consume 1 nJ per 32-bit instruction, and some RF data transfers are relatively power-hungry, e.g. B. Bluetooth radio frequency (RF) communication chips consume about 100 nJ per transmitted bit, therefore lower power data transmission is required. HF also poses another challenge as there is very limited space for the antennas, resulting in an extremely short wavelength, i. h High frequency data transmission is required.

Microactuation and detection: MEMS

197

Some common applications of MEMS sensors are as follows: Accelerometers can be used in almost all automobiles today to control the deployment of safety airbags. Accelerometers detect the rapid negative acceleration of the vehicle to determine when a collision will occur and the severity of the collision. Angular rate sensors and gyroscopes can be used to measure the rate of rotation or angular velocity of an object. Compared to classical gyros, which are based on optical or (macro)mechanical principles, they do not need a fixed point for reference, are very economical and can withstand harsher environments. The functional principle is based on the Coriolis effect. When a microelectromechanical system (MEMS) resonator is driven at about 10 kHz, the Coriolis force causes a second oscillation perpendicular to the first due to angular velocity. This oscillation is proportional to the speed of rotation and is measured by capacitive methods. Applications include image stabilization and orientation in devices such as cameras and cell phones, as well as navigation systems used in game consoles.

6.4.4 Smart surfaces, skin, paint, matter and dust MEMS can be permanently attached to a solid substrate and form smart surfaces or be more self-contained and form smart structures that can be rearranged. An example of a smart surface is a paint that can detect vibrations because it is loaded with a fine powder of a piezoelectric material called lead zirconate titanate (PZT). When PZT crystals are stretched or compressed, they generate an electrical signal that is proportional to force (Berlin and Gabriel, 1997). MEMS could be mixed with a variety of bulk materials, such as paints, gels, and spread on surfaces or embedded in surfaces or dispersed in other media, such as air and water, and transported as part of them. For example, coating bridges and buildings with smart paint could detect and report traffic and wind loads, and monitor structural integrity. A smart coat of paint on a wall could detect vibrations, monitor the facility for intruders, and suppress noise (Abelson et al., 2000). Smart surfaces can also be woven from organic polymers with conductive and light-emitting properties. Organic computing can be used to form intelligent skins and clothing (Section 5.3.4.3). Like sensor networks, MEMS can also be connected in MEMS networks. The Smart Dust project, led by Kris Pister, produced prototypes of many novel types of networked low-power MEMS sensors (Section 2.2.3.2). In the Claytronics project,10 Goldstein et al. (2005) have proposed the use of thousands to millions of MEMS sensors, actuators, and locomotion devices that behave like malleable programmable matter and can replicate artifacts for a variety of shapes and physical objects. A long-term goal of such MEMS arrays is to be able to assemble them into any 3D shape to achieve a synthetic reality. Unlike virtual reality or augmented reality, synthetic reality allows users to experience synthetic reality without sensory augmentations, such as head-mounted displays, allowing users to physically interact with every object in the system in a natural way. The idea of programmable matter, introduced in the Claytronics project, uses the reassembly and use of moving electronic devices for communication to create new forms of matter. Perhaps the ultimate in programmable matter is to base it on nanotechnology to build matter at the molecular level and move molecules around, not just electrons. Others refer to such irregularly distributed ensembles of computing particles on a surface or in a volume where the individual particles have no a priori knowledge of their positions or orientations as amorphous computing and sprinkle computing (Zambonelli et al., 2005). These particles can be glitched, contain sensors and perform actions, and in some applications they can be mobile and are known as amorphous computational particles (Abelson et al., 2000).

The Claytronics project is a collaborative effort between researchers at Carnegie Mellon University and Intel Research. Website: http://www.cs.cmu.edu/claytronics/, accessed February 2008.

198

Mark, register and control

New design and manufacturing models are required to build such systems (Section 6.4.1). Novel techniques are needed to manage sets of MEMS devices, possibly by incorporating behaviors based on self-organizing interaction mechanisms (Section 10.5.1). However, there are still security issues affecting these management techniques that limit their effectiveness in managing sparse MEM collections in practice.

6.4.5 Downsizing to nanotechnology and quantum devices Gordon Moore (1965), co-founder of Intel, made what is now commonly known as Moore's Law, which predicts that the number of transistors on an IC chip will double approximately every two years .11 This does not mean, of course, that the processing power of the software is also increased in this way. Graham (1989) identifies two reasons why software performance may not increase at this rate: it is not just an increase in transistor density, the computational architecture may also need changes to take advantage of it, and this can take 5 to 10 years. Furthermore, communicability does not necessarily increase in proportion to the increase in computation. In terms of hardware, this kind of increase in transistor density on IC chips is typically made possible by advances in photolithography that come every six or seven years, allowing for three IC size reductions at once. The shorter the wavelength of light used, the thinner you can make the mask, and the smaller you can make the transistor parts and wiring. However, more recently, problems with using the latest innovative technique, known as extreme ultraviolet photolithography, to produce light at wavelengths shorter than 13.5 nm can cause delays between the end of the sun's UV spectrum and the beginning of X-rays, resulting in a transistor surge. density (Santo, 2007). Also, reducing the size of the transistor and increasing the density of the transistor is much more complex than simply reducing the optical wavelength used to draw the circuit mask. Bohr et al. (2007) found that the thin layer of insulation that electrically insulates the gate of the transistor is only a few atoms wide and has problems when the insulation is broken. Therefore, it is necessary to model and design new materials at the molecular level, and this is a key aspect in the field of nanotechnology. Nanotechnology12 can be defined as manipulation, precise positioning, measurement, modeling and fabrication to create systems below 100 nm (Poole et al., 2003). Nanotechnology is not just smaller MEMS. It also relies on a broader range of materials and mechanisms and sizes down to the molecular level, e.g. B. Carbon-based nanotubes have much better conductivity than silicon-based semiconductors at the nanometer level (Banerjee and Srivastava, 2006). In contrast, MEMS focuses on semiconductor-based single IC chip technology and micromachining. While MEMS tends to use a top-down approach to device design, nanotechnology also tries to use a bottom-up design. The drive to make transistors switch faster and use less power was to make them smaller. However, as electronic components approach nanometer size, strange things start to happen as electrons begin to reveal their quantum nature, e.g. B. Electrons have the potential to pass through a transistor even when it is turned off. This prompted early concern about the viability of nanotechnology arising from the quantum uncertainty of whether or not it would make these systems impossible to be unreliable. A second major limitation is thermal noise, which causes local molecular motion due to heat. This phenomenon limits what can be done mechanically.

11 There is some controversy as to whether Moore's First Law applies every two years or every eighteen months. There is also Moore's second law, which states that the cost of doubling the circuit density increases in accordance with Moore's first law. 12 Nanotechnology overlaps with the terms nanocomputer, molecular nanotechnology, and molecular manufacturing.

Embedded systems and real-time systems

199

molecular scale (Peterson, 2000). In addition, progress must also be made in the positioning and control of structures at this level (Devasia, et al., 2007). Nanotechnology first proposed using a bottom-up design approach to be able to assemble custom molecular structures for specific applications, such as creating rigid molecular materials to reduce the effects of thermally induced molecular motion. A major challenge for this design process is the complexity and novelty of understanding and being able to model materials at this level. More research is needed to understand how combinations of materials, particularly composites, confer specific physical and functional properties to materials at the molecular level. Chang (2003) provides an overview of some built nanotechnology devices. Nanotechnology requires two main types of technical assistance: molecular positional arrangement, manipulation and positioning of individual atoms, and massive parallelism; otherwise a robotic arm would take forever to build an object the size of a kg molecule for molecule, a large number of robotic arms in parallel would require work.

6.5 Embedded Systems and Real-Time Systems Embedded systems are primarily used online to perform tasks in the physical world, unlike general-purpose computers, which are often used offline to access and share information. Therefore, embedded computing systems differ from general purpose systems (MTOS) in three important ways. Embedded systems focus more on the implementation of individual tasks. Security criticality can be important because actions affect the physical world. Third, tasks often need to be scheduled with real-time constraints in mind. An embedded system is a component of a larger system that performs a single dedicated task. This can use a much simpler and cheaper operating system and hardware, since there is only one process. This simplifies memory management and process control, and eliminates inter-process communication that normally must be supported in an MTOS. An embedded system may or may not be visible to a user of that system as a computer. It may or may not have a control interface visible to human users. Embedded devices can be local and fixed, e.g. a printer or an AV recording or playback device, or it can be mobile and distributed on planes, ships, and on Internet devices. An embedded system is programmable and contains one or more dedicated task computers, microprocessors, or microcontrollers. A microprocessor is an integrated circuit that forms the central processing unit of a computer or embedded controller, but requires additional support circuitry to function. A microcontroller is a microprocessor plus additional types of processors that support other embedded devices in a single package. Other device types supported by devices may include serial (COM) ports, parallel ports, USB ports, Ethernet ports, A/D and D/A ports, interval timers, watchdog timers, event counters/timers, real-time clocks (RTC) . , digital signal processing (DSP) and numerical coprocessors. Even a general purpose computer can contain additional dedicated computers for controlling battery charging and discharging and for AV recording and playback. Embedded systems can contain programmable logic elements, such as Field Programmable Gate Arrays (FPGAs) or application-specific integrated circuits (ASICs), as opposed to non-programmable processing chips, such as CPUs. Single process embedded systems generally do not connect to other external systems without exposing and redesigning their control interfaces; it is difficult to connect them to be part of a larger system. Embedded computer systems differ from an MTOS computer in several ways. They use a wide range of microprocessors, hundreds of types dedicated to a specific task or tasks, e.g. For example, peripherals, networks, etc. Almost all embedded designs (hardware and software) are unique. Each embedded computing device can function within its own strictly defined operational limits, e.g. B. the heating system is designed. Designs are often developed for the highest possible performance at the lowest cost. Performance may not be a major consideration. They often operate under moderate to severe real-time constraints. These properties suggest that interoperability and

200

Mark, register and control

Networking embedded system devices can be more challenging than MTOS computing devices. Software bugs can have life-threatening consequences when systems fail. Therefore, some types of embedded systems are designed to be fault tolerant. Fault tolerance on embedded systems can be a thousand times better than on desktop computers. Embedded systems often have power consumption limitations. They are often designed to work in a wider range of physical environmental conditions than a personal computer, e.g. B. in humid, hot and cold and dark conditions. They often use fewer system resources than desktop or laptop systems. There may be no hard drive or removal media. All codes may need to be stored in the ROM. Embedded systems may or may not have some general operating system (OS) services available, e.g. B. There is no standard C language printf() function for debugging when there is no terminal. Therefore, embedded systems may require special development tools, e.g. B. On-chip debugging facilities can be used instead of printf(). Embedded computer systems are not always easy to program because they may not separate the hardware from the control software that is critical to making the network more programmable. Hardware and software are highly vertically integrated and interdependent, like ASICs, microcontrollers. There are many examples of the use of embedded system applications. Transportation vehicles are not only an important application area for the use of new sensor technology and for robots in production and for unmanned self-driving vehicles (robots), but are also a good application area for embedded systems. Modern automobiles interconnect multiple embedded systems for anti-lock braking systems (ABS), cruise control, air conditioning, outside mirrors, locomotion and traction sensor data monitoring, etc. (Leen and Heffernan, 2000). In the ABS subsystem, for example, sensors measure the speed at which the wheels are spinning. If wheel speed drops rapidly, ABS recognizes a risk of lockup and immediately reduces hydraulic pressure to the brake, then increases it to just below the lockup threshold. This relaxation and increase in pressure can be repeated several times per second. The various integrated subsystems must be connected to form a holistic control and monitoring system. In the mid-1980s, Bosch developed the Controller Area Network (CAN), one of the first and still one of the most widespread vehicle networks with more than 100 million CAN nodes sold annually. A typical vehicle may contain two or three separate CANs that operate at different baud rates. A low speed CAN of less than 125 Kbps typically manages a car's convenience electronics for window and seat movement controls. These control applications are not real-time critical and use a power-saving sleep mode. A higher speed CAN performs more critical functions in real time, such as engine management, anti-lock brakes and cruise control. Some embedded systems are networked, but are only connected to certain network services. For example, automobiles and vacuum cleaners can be designed to notify the manufacturer's service center when they malfunction or require service. ATMs, electronic point of sale or EPOS terminals, ticket reservation machines, communication, infotainment, and GPS devices are other examples of embedded connected devices. System support for organizations (robots) and coordination of multiple autonomous process controllers is discussed in more detail in Chapter 9.

6.5.1 Application-Specific Operating Systems (ASOS) Embedded systems require an Application-Specific Operating System (ASOS) that can be customized and reconfigured to meet the needs of specific applications to provide lower cost and higher performance. performance by removing common MTOS features (chapter 3.4.3) that are unnecessary for certain applications and by better tailoring the contained functions (Friedrich et al. 2001). Software or hardware processing techniques, such as the use of data compression, can expand data storage capacity for a given storage capacity, thereby reducing the cost of memory and the power required to access stored data. in memory of.

Embedded systems and real-time systems

201

A key design issue is whether the MTOS configuration is fixed at design time or can be changed during the operational lifetime to accommodate new requirements. For example, an integrated environmental control system for a smart home uses a microcontroller running an ASOS that does not require many of the file system functions, interprocess communication, or controller, network, and security facilities provided by a MTOS. However, adding an intrusion alarm system to the smart home may require network support to connect various security alarm and monitoring components and adding a file system for system logging. Some devices running ASOS may need to be updated. Friedrich et al. (2001) review various component-based operating systems to support this.

6.5.2 Real-time operating systems for embedded systems Real-time applications for embedded systems are a subset of embedded system applications that perform security-critical tasks related to time constraints, as their violation can cause the system to be disabled. unsafe, such as the operation of automobiles, ships, airplanes, monitoring and control of medical instruments, multimedia transmission, factory automation, financial transaction processing, and video game machines. Real-time operating systems, or RTOS, can be viewed as a resource constrained system where primary resources, such as data transmission, are limited in time. This, in turn, limits the number of CPU cycles, but other resources may also be limited, such as: B. Memory. An RTOS responds to external events that interrupt it. The RTOS design focuses on programming efforts to allow processes to comply with real-time constraints and to optimize memory allocation, process context switch time, and interrupt latency (Section 3.4, Li et al. ., 1997). Because multiple interrupt events can occur, an RTOS must have a mechanism to prioritize interrupts. An RTOS can also use additional process control to block certain processes in memory to prevent the process from swapping overhead. There are a number of real-time design considerations to support critical task response time, time to detect an event and trigger the appropriate action to handle it, critical data transfer rates, response time optimization and data transfer rates, and their optimization where there are are simultaneous tasks. The two key factors that affect response time are process context switching (for switching between different processes) and interrupt latency (the time delay before context switching is possible). Timeliness is the most important aspect of a real-time system. A real-time system (RTS) is one in which the timing of a result is as important as the result itself. A correct answer produced too late is just as bad as a wrong answer or no answer at all. An RTS is one in which the correctness of the calculations depends not only on the logical correctness of the calculation, but also on the time in which the result is produced. If the time constraints are not met, this is called a system failure. Time restrictions may vary between different systems in real time. As such, an RTS can fall into one of three categories: soft, hard, or firm. In a smooth RTS, the time requirements are defined using an average response time. A single late arrival calculation is not significant to the operation of the system, although there could be many late arrivals. An example of this is flight reservation systems. If a single calculation is delayed, the system response time may be delayed. The only consequence, however, would be a potential frustrated passenger. In a hard RTS, time requirements are crucial. A late reply is incorrect and will result in system failure. Activities must be completed before a specific deadline because if a deadline is exceeded, the task will fail. Deadlines can be a specific time, a time interval, or the occurrence of an event. This requires the system to be able to predict how long the calculations will take. An example of a hard RTS is a pacemaker. If the system takes longer than expected to start treatment, biological damage may occur. In some types of STRs (fixed), the time requirements are a combination of hard and soft requirements. Typically, the calculation has a shorter soft requirement and a longer hard requirement. On

202

Mark, register and control

An example of a fixed RTS is a patient ventilator that helps patients with respiratory problems breathe for them. The system has to ventilate a patient so many times in a given period of time. A delay of a few seconds in initiating the patient's breath is acceptable, but not more.

6.6 Control systems (for tasks in the physical world) The simplest type of control is only activated when defined thresholds,13 z upper threshold, are exceeded. However, this often causes the temperature to oscillate between the lower and upper thresholds. Feedback control uses continuous monitoring of some outputs using sensors and responds to their changes to regulate the output. There are two basic types of feedback. Negative feedback attempts to reduce some change in a system output or condition, while positive feedback serves to reinforce a system output or condition. There are several basic feedback control designs. Negative feedback can use a derivative of the output to combine with the input to regulate the output. In a simple proportional control system, a signal is negatively fed back to a system proportional to the degree to which the system output deviates from the reference value. This results in a much smoother regulation of a heater around its set temperature and, in the case of a voltage controlled electric motor, allows the speed to be regulated more smoothly.

6.6.1 Programmable Controllers Programmable Controllers were developed to support much more configurable and flexible control, e.g. B. Microcontroller. The hardware architecture of microcontrollers is much simpler than that of general purpose processor motherboards in personal computers. Microcontrollers do not need to have an address bus or a data bus because they integrate all of the RAM and non-volatile storage on the same chip as the CPU. The CPU chip may be a simpler 4-bit processor than 64-bit processors. I/O control may be easier since there may be no video screen output or keyboard input. However, the complexity of microcontrollers can vary, for example they may contain digital signal processors (DSPs). Originally, microcontrollers were only programmed in assembly language and later in C code. Programs are often developed on an emulator on a development platform (PC), which is then downloaded to the target device for execution, maintenance, debugging and validation. Newer microcontrollers can be integrated with on-chip debugging circuitry accessed through an in-circuit emulator that allows a programmer to debug embedded system software. The in-circuit emulator usually has a JTAG interface.14 This is a special four or five pin interface that is added to a chip and is designed to allow multiple chips to connect their JTAG interfaces together. Therefore, only one test probe needs to be connected to a single JTAG interface to access all the chips on a circuit board. Although it is most commonly used to access test-specific logic built into integrated circuits, it can also be used as a Field Programmable Gate Array (FPGA) programming interface (de la Torre et al., 2002).

Threshold control systems are similar to event-condition-action or ECA systems. The JTAG (Joint Test Action Group) refers to the IEEE 1149.1 standard titled Standard Test Access Port and Boundary Scanning Architecture. 14

Control systems (for physical world tasks)

203

6.6.2 Simple PID Controllers There are certain circumstances where the output of the proportional or simple proportional controller will not regulate correctly, eg. B. when there is a long delay in the plant between the change in the input and the response of the output. In this case, a parameter can be regulated below its reference value. To solve this problem, an integral or differential control, or both, can be added to the control. A PID controller is so called because it combines proportional, integral, and derivative control (see Figure 6.9). A proportional controller is simply the error signal multiplied by a constant and sent to a hardware unit. An integral controller deals with the past behavior of the controller. It is almost always used in conjunction with a proportional controller. It adds all previous inputs and is used to give the control loop a long-term precision using the past. A derivative controller is used to predict the behavior of the plant, then this could be used to stabilize the controller via a feedback loop. Such control algorithms, P, PI, PD or PID, are usually simple enough to be used in conjunction with some adaptive controls, e.g. B. the gain of the proportional element, which will be encoded in the controllers.

6.6.3 More complex controllers The PID controllers described in the previous section can be used for rough control of robotic arms, eg. B. to support palletizing and to support roughly controlled locomotion. Controller design algorithms are based on the assumption that some basic knowledge of the plant to be controlled existed and that some approximation models can be used to form a starting point for designing the controllers and then adjusted using feedback to do the job. to become the uncertainties of the system parameters. However, the algorithms are not suitable for tasks that require finer control, dynamic control, handling of control uncertainties, and the need to support adaptive control. There are several main sources of control uncertainty that can arise:

• Environmental dynamics is a stochastic form of naturally changing physical objects; • random action effects: some actions can have different effects due to variations in the physical properties of things when they are formed, propagation of complex effects;

• Inaccurate models: some features of the physical environment, eg. B. the climate, are inherently non-deterministic;

• Estimate: p. B. Nonlinear behavior is often approximated by a linear solution in part of its range;

• Sensor limitations: incorrect sensor placement, partial world view, and poor signal reception, see Table 6.1. Two of the main techniques for controlling insecure systems are adaptive control and robust control. Adaptive control uses online identification, in which parameters of interest to the plant are identified using error predictions (indirect adaptive control) or parameters are monitored and adjusted using tracking errors (direct adaptive control) to try to identify uncertain plant parameters. learning system. Adaptive control is a nonlinear feedback technique that performs identification and control simultaneously in the same feedback loop. This can be used to deal with systems with large uncertainties (Xie and Guo, 2000). The adaptive approach is applicable to a wide range of parameter variations, but is sensitive to unstructured uncertainties. A robust controller must make the system insensitive to all uncertainties and the final controller has a fixed structure. A robust controller is capable of handling small uncertainties (Yu and Lloyd, 1997).

Mark, register and control

204

PID-Regler Proportional

P controller

+ e(t)

∑

Integral

to receive)

Proportional

mi(t)

–

pie)

derived

Control system

reference value r(t)

∑ – f(t)

Rules

Fehler e(t) = r(t) – f(t) Feedback

DAC and drive

ADC

input i(t)

attached file

output o(t)

converter

Figure 6.9 Two simple control systems: a proportional controller (top) and a PID controller (bottom)

6.7 Robots Robots are an important type of application that combines sensor and control technology. A related term is cybernetics, a Greek term meaning self-piloting like a governor or pilot. In the early 1960s, robots15 were used to automate industrial tasks, particularly in manufacturing, to free human workers from repetitive, risky, and damaging tasks and from tasks that required skill, strength, or dexterity beyond their capabilities. of people go further, positioning in z and movement. The main components of the robots are:

• End effector or actuator: a final part of a robot that affects the world in some way, eg. B. gripping, drilling, or stacking, and attached to a swing arm, gantry, or chassis.

• Locomotion in the form of a cart or mobile arm. For example, a robotic arm consists of a chain •

•

made of rigid bodies connected at joints to position its end effector, while a moving chassis may be propelled by wheels or tracks. Propulsion: propelled by air, water pressure or electricity. Power is used to drive controlled motion defined by the number of degrees of freedom of motion. The main electronic component that creates movement is the electric motor. There are different types such as AC motors and DC motors such as servo motors and stepper motors. Controller – Regulates the movement of end effectors based on sensor inputs, motion planning, and control theory, such as B. Drilling or punching, palletizing (stacking and arranging).

15 In fact, the word “robot” derives from the Czech word for forced labor or servitude and was first coined some 40 years before the first industrial use of robots by the Czech playwright Karel Capek in his play “Rossum's Universal Robots”. , which opened in 1997 Prague in January 1921.

robot

•

205

factory products on a transportable frame) physical objects and to control or open ports to control the flow of a liquid or gas. The controllers are programmable, allowing the robots to be configured for a variety of tasks. Controls can be quite simple: they can reflect a fixed pattern in a static environment, or they can be variable due to uncertainty, see Section 6.6.3. Sensors: provide input information about the state of the physical world, i. h to determine the position and orientation and follow the movement of an end-effector and feed it into the controller to adjust the effector's behavior and the state of the world. Robots often use multisensory perception of the state of the controlled process and its environment (Brooks and Stein, 1994).

Robots differ in approach from embedded systems in that the programmable controller for the latter is fixed and task specific. Sometimes, however, it is not so clear to define exactly what is a robot and what is not.16 It seems that many everyday devices, such as washing machines, elevators, various individual car parts, and rotating optical and magnetic media storage devices contain the key components of robots, but they are integrated systems because they can be programmed to perform a task. Commercial floor cleaning robots like the Rumba are also integrated systems within the meaning of this definition. However, since their movement, including collision avoidance, is controlled autonomously, these devices are often referred to as (mobile) robots. The most variable robots include automatic floor cleaners, etc. Garcia et al. (2007) have divided the applications of robots in the physical world into three main dimensions:

• a robotic manipulator or robotic arm: it is a chain of rigid bodies connected by joints that lead to • •

an end effector, e.g. B. Manufacturing robots, medical robots that help doctors perform precision tasks, rehabilitation robots such as certain types of prosthetics, etc. Mobile robots: perform tasks in different places by locomotion, e.g. B. Unmanned air, sea and land reconnaissance robots used to create geographic maps, carpet cleaning robots, surveillance and security robots, etc. Bio-inspired robots: take inspiration from biology for their manipulation and mobility, e.g. B. humanoid robots that can walk, robots that can exchange pets, etc.

6.7.1 Robotic Manipulators A manipulator consists of a chain of rigid bodies connected at the joints in an open kinematic chain. A rigid body can have up to six degrees of freedom of motion (DOF). This includes three translational motions, such as up and down motion (lift), left and right motion (rock), and forward and backward motion (push). It also includes three rotational degrees of freedom: pitch up and down; Turn left and right (yaw) and lean from side to side (roll). Joints can be designed in such a way that they limit some of the freedom of movement. A manipulator uses motion planning to calculate the path of the robot between the current or starting position and the target position. A control algorithm, e.g. B. PID based or adaptive control is used to actually control the trajectory. Kinematic calibration is used to compare the theoretically estimated position of the motion plan with the actually detected position. This can be used to adjust the motion planning for more accurate future use. The contact force on the

sixteen

Joseph F. Engelberger (born 1925) is often referred to as the "father of robotics". Together with George Devol, Engelberger developed the first industrial robot in the United States in the 1950s, the Unimate, which was used by General Motors in 1962. He is reported to have once said, when asked to define a robot: " I can't define a robot, but I know one when I see one" (Carlisle, 2000).

206

Mark, register and control

The end effector of the manipulator may need to be adjusted to make it fit for purpose, eg objects slipping or falling off the arm. Additionally, handlers must cope with variations in the components and objects they handle. Here are two solutions, either using adaptive AI techniques or allowing human operators to stay on top of things to use remote control or send remote commands to a local keyer.

6.7.2 Mobile robots Mobile robots use different types of locomotion systems to move based on propellers or screws in air and aquatic environments and based on capillaries, wheels and legs in terrestrial environments. The easiest types of mobile robots to control follow a predetermined path in a static, controlled environment. In dynamic, non-deterministic environments, control is more complicated since the robots must be able to navigate to avoid obstacles. Also, the number of degrees of freedom is much lower compared to a robotic manipulator. In its simplest form, ground vehicles can have DOF, forward and backward. An easy way is to get past obstacles by using collision detection and then backtracking or choosing a random direction, but this can make it inefficient to reach a goal. A more complex approach is to anticipate and avoid collisions and re-route to reach destinations. One of the best known and most successful applications of mobile robots has been Mars Explorer Robots, whose development began in 2000, entered the market in 2003, and landed on Mars in 2004 to complete its mission (Erickson, 2006). One of the most successful task-based, consumer-facing mobile robots since 2002 is iRobot's Roomba robot vacuum, which includes odometry determination to estimate its position traveled relative to an initial location, as well as shock sensors and sensors. IR to detect IR. tags used to track Roomba's movement at the boundary of space (Tribelhorn and Dodds, 2007). Location is used to determine the position of a robot in relation to its physical environment. The location can be local or global. Global localization is covered in more detail in Section 7.4. The local locator is a robot that corrects your position in relation to your home or other current reference locations.

6.7.3 Biologically Inspired Robots The basic components of a robot described in Section 6.5.1 can be compared to human components. Humans use end effectors in the form of their hands, feet, and other parts of their bodies. A muscular system is used for locomotion in order to move the entire body or parts of it. A brain system is a controller that processes sensory information and tells muscles what to do. A propulsion system based on stored energy derived from regular food intake is used to propel the human body. Five basic types of sensors on the surface of the human body are used to see, hear, touch, taste, and smell to gather information about their environment. In general, current mechanical design lags far behind the performance of biological systems. An 80 kg person consumes about 100 W of power, and carrying a 45 kg bag of cement consumes about 350 W of power. By contrast, a "spot welding robot" capable of carrying a 45kg payload weighs 450kg and consumes 5 to 10KW. "Horsepower" (HP) (electrical) can be defined as 750 W, and for comparison, humans can produce around 0.1–0.3 HP over periods of several hours or more. Machines can outperform humans, but humans have a fairly efficient power-to-weight ratio (Carlisle, 2000). Bio-inspired robots are more complex, combining legged locomotion abilities with robotic manipulators. These robots have two main focuses: legged locomotion and human-robot interaction (Garcia et al., 2007). The use of legs allows robots with legs to move more efficiently over uneven ground, stairs, loose, compared to mobile robots.

robot

207

and sandy terrain and over ditches. Bipedal robots often have more DOF than the mobile robot or manipulator robot, making them more complex to control. For example, on the lower body only, you can have six DOF on each leg, three DOF on the hip, one DOF on the knee, and two DOF on the ankle. A particular design challenge for bipedal robots is stability, for general balance, a larger body mass compared to the mass of the legs, and in the use of leg and body movements, steps in which that the individual legs are not constantly in contact with the ground at any point. Bipedal robots often aim to control this balance by controlling the zero moment point (ZMP) so that the dynamic reaction force when the foot makes contact with the ground does not create moment or rotational force. A second approach is human-robot social interaction (Fong et al., 2003). Robots can help humans by perceiving situations of interest to specific human groups, such as Robots can also fulfill a social role: in relation to interaction with humans at the level of emotions (affective computing), e.g. B. artificial pets; socially guided learning through imitation or guidance (individual and safe instruction) and through interaction with more human-oriented interfaces, such as speech recognition.

6.7.4 Nanobots Nanobots can be made in the same way as MEMS, see Section 6.4.1, or at the molecular level. Although the microscopic world obeys the same physical laws as the macroscopic world, the relative importance of physical laws in terms of how they affect mechanics and electronics at this level changes and this must be taken into account (Abbott et al., 2007 ). The relationship between volume (Length3) and surface area (Length2) changes when an object is scaled, since they depend differently on length. In general, electrostatic forces are assumed to scale well on the microscale, magnetic effects seem to scale well or not on the scale, and fluid mechanics becomes simpler as the effect of turbulence becomes less important, e.g. B. The flow pattern does not change appreciably whether the flow is slow or fast and the flow is reversible (Abbott et al., 2007). The nature related to microorganisms can be used to provide nanobots with a host body in which to move. Bacteria contain a magnetic z field (Martel, 2006). Reducing the size of the device to these nanoscale dimensions comes with many interesting challenges, as z and Nelson, 2007).

6.7.5 UbiCom Robotic Application Development Most types of industrial robots, although programmable, are expensive and too specialized to perform only a few tasks and are designed to work in clean, static environments. They have precise actuators and sensors to determine their position, perform precise movements, and operate robotic effectors in carefully controlled physical environments. Most low-cost consumer robots, such as robotic pets, while capable of sensing the physical environment and sensing and adapting to simple interactions with humans, tend not to be easily programmable by consumers. Various robot toolkits are available, allowing experienced developers to develop their own applications. To illustrate some of the problems with using programmable robots, an overview of using a programmable robot to automate changing a part of the physical world on behalf of a human owner is provided. Because resources and costs are limited, a low-cost robot

Mark, register and control

208

Light sensor

Motor A

ultrasonic sensor

Motor B

CPU-Motor C

Figure 6.10 Using the Lego Mindstorm NXt robot to solve Rubik's cubes

The Lego Mindstorm NXT Robot17 Toolkit (Ferrari and Ferrari, 2007) was chosen as the platform. This consists of a dual processor board that allows one coprocessor to be dedicated to a task such as monitoring an optical tachometer to determine the rotation of a servo motor, while the main processor performs other tasks. A basic kit has three servo motors and four sensors, a simple touch sensor, a light sensor to measure light intensity, a sound sensor, and an ultrasonic sensor to determine distance. The chosen task was to have the robot manipulate a Rubik's cube (Figure 6.10) (a mechanical puzzle invented by Erno Rubik in 1974) to its final state in which all sides show a single color (the solved state). The goal could be to delegate all this task to the robot or to design the robot to interact with humans, give advice to humans and show them how to make certain modifications to the cube. Both objectives include several subtasks that must be designed. First, the robot mechanics must be designed so that it can hold the cube in such a way that the robotic arm can rotate the entire cube or just one side of the cube, and move the optical sensor through the cube in turn to change. the color of the sides of the to determine the dice. Second, a design is needed for how and when the robot perceives the state of the world, in this case the cube. In order to reduce the amount of detection and movement of the sensors on the whole face, it can be decided to start detecting the state of the cube only once. The sensors could also be used to indicate that the cube is correctly positioned within range of the robot. A robot could feel your guts

17 Home page http://mindstorms.lego.com/, accessed November 2007. There are several third-party programming languages and environments, such as Java for Lego Mindstorms http://lejos.sourceforge.net/, accessed November from 2007.

robot

209

status to later assess if you have enough energy to complete the task. The state of the cube also needs a representation that a cube solving algorithm can understand. Third, a scheduling algorithm is needed to perform a series of rotations of the cube and its cube sides to get the cube in the solved state. There are approximately 43 million million million permutations for the cube positions that must be searched for a brute force search (Section 8.7.1) to produce a plan. The use of informed searches, such as B. a corner cube algorithm first, but can drastically reduce searches. For example, if there are two opposite faces with different colors, there is no vertex or edge that contains those two colors. The derived plan should define the cube rotation actions that the cube can practically perform, and this may need to take into account the fixed reserve of energy that the robot has. Finally, an overarching architecture is needed that integrates these various subtasks, such as B. the one-step vertical hybrid architecture (Section 8.3.7). It uses a lower reactive layer to take in sensory information about the world and an upper layer of search and planning reasoning to derive the plan for solving the cube. All the software to implement these tasks must then be programmed and downloaded to the robot's processor so that it runs when it is activated. There are several practical problems that arise when physical robots perform tasks. The sensitivity of the optical sensor is limited and may not be enough to distinguish light intensity between some different colors like orange and red. If the cube is not positioned correctly on the robotic arm or device, the sensors will detect gaps and misinterpret them as the state of a part of the cube. The dice have different friction values to turn on different sides. In some cases, the robot may only turn a cube halfway and stop when it detects too much friction. Also, due to the Lego bricks, there is some elasticity in the robotic arm, which means that parts of the robot may fall off when the robotic arm moves. The mechanics of the robot require maintenance. The robot arm's handle can slide when the robot is holding the cube, allowing one side of the base to rotate. Therefore, a lot of low-level effort is required to tell the robots to perform certain tasks. Tasks must be carefully designed to match the capabilities of the robot. In the real physical world, the robot must be able to deal with incorrectly applied physical forces and incorrectly detected states of the physical world. Flexible, general-purpose UbiCom robots that can function as autonomous assistants or servants for mass human use do not yet exist.

EXERCISES 1. Does location determination involve a tag or a survey? Explain. 2. The annotation or labeling of physical objects can be classified in two dimensions: whether the user of the annotation shares a location (on site) with the physical object or not; when the annotation anchor is attached directly to the object it refers to, as opposed to detached annotation. Explain the effects and give some usage examples. 3. Compare and contrast the Semacode scheme with local interaction in the physical world with other schemes such as different types of RFID tags and the Cooltown project (Section 2.2.2.4). 4. The IPv6 address is huge. Is it big enough to uniquely identify every living thing on earth? Suggest schemes that we could use to try to label every living thing on earth. What would be the pros and cons? 5. Discuss how we can create and organize an address space to identify all useful man-made objects in the physical world. Compare and contrast the EPC identification scheme to identify things with the IPv6 addressing scheme. Could EPC be used instead of IPv6 on the Internet to represent virtual objects alongside physical objects? 6. Compare and contrast tags with sensors.

Mark, register and control

210

TRAINING

(Continuation)

7. Discuss the address space layout and routing algorithms used in sensor networks. 8. Discuss why the MEMS (micro) design of a mechanical device is so different from the equivalent macro design. 9. Embedded systems generally do not require a full general-purpose operating system to function. Why not? 10. Define the main elements of a robot. Define the properties that characterize a robot. Define three main types of robots. How is a robot different from an embedded system and a feedback control system? (More exercises are available on the book's website.)

References Abbott, J.J., Nagy, Z., Beyeler, F., et al. (2007) Robotics in miniature. Part I: Microrobotics. IEEE Robotics and Automation, 14(2): 92-103. Abelson H, Allen D, Coore D et al. (2000) Amorphous computing. ACM Notices, 43(5): 74-82. Akyildiz, I.F., Su, W., Sankarasubramaniam, Y. et al. (2002) A survey of sensor networks. IEEE Communications Magazine, 40(8): 102-114. Al-Karaki, J.N. and Kamal, AE (2004) Routing Techniques in Wireless Sensor Networks: A Survey. IEEE Wireless Communications, 11(6): 6-28. Banerjee, K. and Srivastava, N. (2006) Are carbon nanotubes the future of VLSI interconnects? Proceedings of the 43rd Annual Design Automation Conference, San Francisco, USA, pp. 809–814. Berlin, A.A. and Gabriel, K.J. (1997) Distributed MEMS: new challenges for computing. IEEE Computer Science and Engineering, 4(1):12-16. Bohr, MT, Chau, R.S., Ghani, T. et al. (2007) The High-K Solution. IEEE Spectrum, 44(10):23-29. Bonnet P, Gehrke J and Seshadri P (2000) Querying the physical world. IEEE Personal Communications, 7(5): 10-15. Brooks, A. and Stein, L.A. (1994) Building brains for bodies. Autonomous Robots, 1(1): 7-25. Bush, V. (1945) As we can think. The Atlantic Monthly, Vol. 176: 101-108. Reprinted and discussed in ACM Interactions, 3(2), March 1996: 35-67. Buttazzo G. (2006) Research trends in real-time computing for embedded systems. ACM SIGBED Revision 3(3): 1-10. Carlisle, R. (2000) Robot Mechanisms. In Proceedings of the 2000 IEEE International Conference on Robotics and Automation, ICRA 2000, San Francisco, USA, pp. 701-708. Chang, C-Y. (2003) Highlights in the nano world. IEEE Proceedings, 91(11): 1756-1764. De laTorre E, García M, Riesgo T et al. (2002) Non-intrusive debugging using the JTAG interface of FPGA-based prototypes. In Proceedings 2002 IEEE International Symposium on Industrial Electronics, ISIE 2002, 2:666-671. Devasia, S, Eleftheriou, E, Moheimani, (2007) S.O.R. A review of control problems in nanopositioning. IEEE Transactions on Control System Technology, 15(5): 802-823. Dong, L. and Nelson, B.J. (2007) Tutorial - Small-Scale Robotics Part II: Nanorobotics. IEEE Robotics and Automation, 14(3): 111-121. Erickson, J. K. (2006) Living the dream: an overview of the Mars exploration project. IEEE Robotics and Automation, 13(2): 12-18. Estrin, D., Culler, D., Pister, K. (2002) Connecting the physical world with ubiquitous networks. IEEE Pervasive Computing, 1(1):59-69. Feiner, S., Macintyre, B., Seligmann, D. (1993) Knowledge-based augmented reality. ACM Notices, 36(7): 53-62. Ferrari, M. and Ferrari, G. (2007) Building robots with LEGO Mindstorms NXT. ISBN-13 978-1-59749-152-5. Fong, T., Nourbakhsh, I., Dautenhahn, K. (2003) An investigation of socially interactive robots. Robotics and Autonomous Systems, 42:143–166.

references

211

Friedrich, L.F., Stankovic, J., Humphrey, M., et al. (2001) A review of component-based configurable operating systems for embedded applications. IEEE Micro, 21(3): 54-68. Frost, GP (2003) Evaluation of smart dust. Computing in Science and Technology, 5(6): 6–9. Fuller, SB, Wilhelm, EJ and Jacobson, JM (2002) Inkjet-printed microelectromechanical nanoparticle systems. Journal of Microelectromechanical Systems, 11(1):54–60. Garcia E, Jimenez MA, De Santos PG et al. (2007) The evolution of robotics research. IEEE Robotics & Automation Magazine, 14(1):90-103. Gemmell, J., Bell, G., & Lueder, R. (2006) MyLifeBits: a personal database for everything. ACM Notices, 49(1): 88-95. Gershenfeld, N. (1999) The Personal Maker. In When things begin to think. London: Henry Holt & Co, pp. 63-75. Goldstein, S.C., Campbell, J.D. and Mowry, T. C. (2005) Programmable matter. Computer Science, 38(6):99-101. Graham, JH (1989) Special Computer Architectures for Robotics: Tutorial and Survey. IEEE Transactions on Robotics and Automation, 5(5): 543–554. Hansen, FA (2006) Pervasive Annotation Systems: Technologies and Challenges. In Proceedings of the 17th Conference on Hypertext and Hypermedia, Odense, Denmark, pp. 121-132. Heidemann J, Silva F, Intanagonwiwat C, et al. (2001) Creating efficient wireless sensor networks with low-level names. In Proceedings of the 18th ACM Symposium on Principles of Operating Systems, Banff, Canada, pp. 146–159. Holldobler, B. and Wilson, E.O. (1990) The ants. Cambridge, MA: Harvard University Press. Kahn, J.M., Katz, R.H. and Pister, K.S.J. (2000) New challenges: mobile networks for Smart Dust. Communications and Networks Magazine, 2(3): 188–196. Kaiser, WJ, Pottie, GJ, Srivastava, M. et al. (2004) Networked infomechanical systems (NIMS) for ambient intelligence. In W. Weber, J.M. Rabaey and E. Aarts (eds.) Ambient Intelligence. Berlin: Springer Verlag, pp. 83–114. Koster, DA, Markus, KW. and Walters, MD (1996) MEMS: Small Machines for the Age of Microelectronics. Computer Science, 29(1):93-94. Krueger, MW (1993) Environmental engineering: The virtual world of the real world. ACM Notices, 36(7): 36-37. Kumar, R., Farkas, K.I., Jouppi, N.P. et al. (2003) Heterogeneous Single-Core ISA Architectures: The Potential for Reduced Processor Performance. In Proceedings of the 36th Annual IEEE/ACM International Symposium on Microarchitecture, Washington, DC, USA, pp. 81–92. Leen G and Heffernan D (2000) Expansion of automotive electronic systems. Computer Science, 35(1):88-93. Li, Y., Potkonjak, M. and Wolf, W. (1997) Real-time operating systems for embedded computing. In Proceedings of 1997 International Conference on Computer Design (ICCD'97), pp. 388-392. Mackay, W. E. (1998) Augmented reality: bridging real and virtual worlds: a new paradigm for interacting with computers. In Proceedings of the Advanced Visual Interfaces Workshop (AVI'98), pp. 13-21. Mansour, R.R., Bakri-Kassem, M., Daneshmand, M., et al. (2003) RF MEMS Devices. In the process. International Conference on MEMS, NANO and Intelligent Systems, pp. 103–107. Meng, E. and Tai Y-C. (2003) Polymer MEMS for microliquid delivery systems. Paper presented at the ACS Polymer MEMS Symposium, New York, USA, pp. 552–553. Martel, S. (2006) Toward magnetic resonance-guided MC-1 magnetotactic and ferromagnetic bacterial carriers for targeted therapies in tumor angiogenesis-stimulated arteriolocapillary networks. In Proceedings of the IEEE International Conference of the Society for Engineering Medicine Biology, pp. 3399-3402. Moore, G. E. (1965) Stuffing of more components in integrated circuits. Electronics, 38(8):114-117. Nath, B., Reynolds, F., and Want, R. (2006) RFID Technology and Applications, IEEE Pervasive Computing, 5(1), 22-24. Norman, D. A. (1988) The psychology of everyday things. New York: Essential Books. Petersen, K. (2005) A new era for MEMS: Solid-state sensors, actuators, and microsystems. In Digest of Technical Papers. The 13th International Conference on Transducers '05, 1:1-4. Peterson, C. (2000) Taking technology to the molecular level. Computer Science, 33(1):46-53. Poole, CP, Jones, FJ & Owens, FJ (2003) Introduction to nanotechnology. Chichester: John Wiley & Sons, Ltd. Rebeiz, G. M. (2003) RF Mems: theory, design and technology. Chichester: John Wiley & Sons, Ltd.

212

Mark, register and control

Santo, B. (2007) Plans for next-generation chips in jeopardy. IEEE Spectrum, 44(8): 8-11. Shen, C-C., Srisathapornphat, C. & Jaikaeo, C. (2001) Sensor Information Network Architecture and Applications. IEEE Personal Communications, 8(4): 52-59. Tanenbaum, A. S. (2001) Modern Operating Systems, 2nd Edition, Englewood Cliffs, NJ: Prentice-Hall. Tribelhorn, B. and Dodds, Z. (2007) Roomba evaluation: a ubiquitous and inexpensive platform for robotics research and education. Paper presented at the IEEE International Conference on Robotics and Automation, pp. 1393-1399. Wallich, P. (2007) Deeply superficial. IEEE Spectrum, 44(8):56-57. Want, R. (2006) Introduction to RFID technology. IEEE Pervasive Computing, 5(1):25-33. Xie, L-L. and Guo, L. (2000) How much uncertainty can be managed through feedback? IEEE Transactions on Automatic Control, 45(12): 2203-2217. Yeow, T-W., Law, KLE, & Goldenberg, A. (2001). Optical MEMS switches. IEEE Communications, 39(11):158-163. Yu, H. and Lloyd, S. (1997) Adaptive control of variable-structure robotic manipulators. IEE Process and Control Theory and Applications, 144(2): 167-176. Yu, H., Agrawal, D., and El Abbadi, A. (2007) MEMS-based relational database storage architecture. VLDB Journal, 16(2):251-268. Zambonelli, F., Gleizes, M-P., Mamei, M., & Tolksdorf, R. (2005) Computer spray: investigations into self-organization. Mobile and Pervasive Computing, 1(1): 1–20. Zhao, F. and Guibas, L. (2004) Wireless Sensor Networks: An Information Processing Approach. New York: Morgan Kaufmann.

7 Context-aware systems 7.1 Introduction Context-aware systems are systems that are aware of their situation (or context1) in their physical, virtual (ICT) and user environment and can somehow adapt the system to it by benefiting from their knowledge. of this situation. For example, in the Personal Memories scenario, the camera can detect the distance from the camera to the subject of the photo and automatically adjust the focus of the camera lens when the photo is taken. The term context-aware was first used by Schilit and Theimer in 1994 to refer to a system that can provide context-aware information and services to users and applications.2 Schilit et al. (1995) also defined a context-aware system as one that adapts to the context. There are many other similar definitions. For example, Dey (2000) defines a system as context-aware when it uses context to provide the user with relevant information and/or services, where relevance depends on the user's task. Ryan et al. (2008) define a context-aware system as the ability to recognize and sense, interpret, and respond to aspects of a user's local environment and the computing devices themselves. There are several definitions of context. Some are more concrete, others more abstract. Dey and Abowd (2000) define context as “any information that can be used to characterize the situation of an entity that is considered relevant to the interaction between a user and an application”. An example of a more concrete definition of context is to define it as a member of the set of context types, such as B. Location, identities of nearby people, objects, and changes to these objects (Schilit and Theimer, 1994). There are many possible useful sets of contexts, which often depend on applications' use of the context. Environmental monitoring uses various types of distributed sensors to provide environmental context such as air pollution, temperature,

1 Context is often used casually as a synonym for situation. The notion of context is modeled more specifically here. A context is defined here in terms of associations between the states of three types of system environments defined in Chapter 1, physical, human (user) and virtual (ICT), generally relative to a target context. 2 This preceded notable applications that used the concept of context awareness but did not explicitly label this usage as such, e.g.

Stefan Poslad

214

Context Aware Systems

Humidity. Time, place, and person can be combined into a daily context for a personal organizer app.

7.1.1 Chapter Overview This chapter discusses a number of context-sensitive applications. Context-aware applications can be classified in two ways (Section 7.1.2): in terms of the type of environmental context they are aware of, e.g. B. place, time, person, ICT system, etc., and in terms of how the applications fit into the context. Models and architectures for context awareness are discussed in Section 7.2. Several main types of contextual applications are considered below to demonstrate the usefulness of contextual awareness in practice: mobile user awareness (Section 7.3), spatial or local awareness rendering (Section 7.4), temporal awareness rendering (Section 7.5 ) and knowledge of ICT systems (section 7.6). Other chapters also deal with context awareness in the following way. Context awareness is a type of application that uses sensor-based systems (Section 6.3) and can be used to control and regulate the physical environment (Section 6.6). Context-aware system designs can be based on intelligent systems (Section 8.3). Awareness of the external context of the environment is often usefully combined with awareness of internal systems or self-awareness (Section 10.3).

7.1.2 Context-aware applications There are several reviews of systems and applications by Chen and Kotz (2000) and by Baldauf et al. (2007). Loke (2006) provides a good introduction to many types of context-aware systems, including: mobile services; context-aware artifacts: everyday objects and portable devices that can be turned into context-aware, context-aware virtual environments, such as Gelernter's abstraction of a mirror world (Gelernter, 1992). Some of the most common applications such as B. Context-aware mobile systems use multiple types of context awareness at multiple levels (Section 7.3). brown et al. (2000) classify context-aware applications into six types: proactive triggering, interaction optimization, past event recall, future context recall, behavior pattern optimization, and experience sharing. Here, context-aware applications are discussed in terms of the type of context used and how it is applied. One of the dominant applications of contextual awareness since Want et al. (Section 2.2.1.2) was the perception of location for mobile users. Examples of such projects are the Cyberguide project (Long et al., 1996), the GUIDE project (Davies et al., 1999), SmartSight and the CRUMPET project (Section 7.3.4). Cyberguide is a system for providing tourist information to visitors to the Georgia Institute of Technology campus.3 Cyberguide users carry an Apple handheld computer connected via infrared to a central PC to transmit beacons around the center. Lancaster University developed GUIDE to provide tourist information on a Fujitsu handheld. Carnegie Mellon University has a tourist assistance system called SmartSight (Yang et al., 1999). This system uses a handheld computer to answer spoken questions about local points of interest, provide translation, and aid in navigation. There are many other human activities and automatic services based on physical phenomena. Here are some examples. Conveyor systems can automatically brake when

Abowd's group at the Georgia Institute of Technology has been particularly active in leading context-awareness research, conducting research on context-aware applications from the mid to late 1990s. Their projects include Classroom 2000 (Section 2.2.2.1) , Aware Home, launched in 1998 and still ongoing, and Cyberguide.

Introduction

215

recognize obstacles ahead (collision detection). Weather-aware apps include outdoor sports like boating, farming, and trading. Supermarkets use weather forecasts to determine the amount of certain foods to stock, since certain foods, such as salad ingredients, berries and grilled meat, are consumed more when it is sunny. Spatial awareness is included as a basic type of awareness of the physical environment, since objects are inherently located in space (and time). Spatial awareness allows spatial information, such as maps, to be related to the current location and destination of its users. Time-aware apps schedule their services in terms of time. Personalized systems allow services to be tailored to users, based on their personal preferences, e.g. B. Users can get restaurant recommendations based on their food preferences. A second way to classify context-aware applications is into passive versus active context-aware subtypes.4 In a passive context-aware system, context is presented to users to alert them to the current context or a change in context. The system is not active to adapt a use or application to the context. In an active context-aware system, context adaptation is performed by the UbiCom system, not by human users. Two main types of passive context-aware applications are simple context representation of context information and context-based tagging. The three main types of active context-aware applications are context-based feedback and task filtering, context-based task triggering, and context-based task control and throttling. A simple contextual representation of postprocessed information involves looking at the value of some variable physical phenomenon, e.g. B. temperature over time and places. A manual decision can be made to trigger an action if the value crosses or stays above a threshold. Some contexts are not always machine-readable, e.g. B. the level of an analog thermometer and needs some post-processing to be useful. The absolute temperature should be equated to a graduated scale of temperature levels. Contextual labeling or annotation of objects in the physical environment refers to the registration of views of physical objects, e.g. B. visual images. Context-based labeling of physical, human, and virtual world entities can improve post-retrieval by processing the labels instead of the virtual views themselves. Tagging can also be used to track objects, including people and property, in real time. The design of context-aware active systems is considered in detail below (Section 7.2.1). Context filtering systems filter information based on context, e.g. B. to automatically adapt content to a mobile user's limited display device and location. Content adaptation can also be useful for less physically capable people who need to adapt information services for users with impaired vision and motor control. Context-aware systems are particularly useful because they can reduce the cognitive load of accessing information, devices, and operating physical devices. This type of information filtering is one of the essential elements of the automated software agent for personal assistants envisioned by Maes (1994). This is because users do not have to manually filter through all possible data to find the data related to their situation, the system can prioritize and select only the data that relates to their context, e.g. B. Show only the maps that contain the current location and the destination location. Context-based task activation focuses on user-centric task activation and provides guidance when subtasks or a task configuration is activated, for example, time in motion. Another example is flood sensors in a river basin that can be configured to trigger a flood mitigation initiative when a certain water level is reached. Real-time capture of physical contexts such as location and

There is also a context-adaptive hybrid system where full active adaptation of the system may require manual operator assistance. The human user guides or corrects the automatic adjustment.

216

Context Aware Systems

Temperature can be used to control the movement of a robotic arm during manufacturing and to regulate the temperature in a heating system. Contextual information systems act as a kind of adaptive distributed event-based system in which contextual events are captured, filtered, and user tasks are adapted to the context. For example, a logistics-type space detection application allows a new route to a destination to be calculated and used when it detects that someone or something has deviated from its predetermined route. Context awareness is an element of control system types in which context is detected to control context within defined operational limits.

7.2 Context-aware system modeling Context-aware system models need to define what a set of contexts describes and how contexts are created, assembled, and used for personalization. Context-aware systems models must define how contexts are represented in computational form and how an operational life cycle can be supported through the use of context-aware systems. Baldauf et al. (2007) have characterized context-aware frameworks in terms of: distributed system architecture, sensor design, context data representation (adapted from Strang & Linnhoff-Popien, 2004's context data classification), type of processing context, which in turn depends on context data representation, context resource determination, historical context data, and security and privacy. The frameworks discussed here share most of these distinctive elements, except that some lack explicit context resource detection, historical context data, and security and privacy support. They also differ in the detailed design and implementation of these functions.

7.2.1 Types of contexts Various ways of classifying contexts have been proposed as follows. Prekop and Burnett (2003) and Gustavsen (2002) refer to types of external and internal contexts similar to these physical and user contexts5. Hoffer et al. (2003) refer to physical and logical contexts, the logical context being similar to the user context. Dey and Abowd (2001) have suggested places such as rooms and buildings, people, whether individuals or groups, and things such as physical objects and components as types of context. Schilite et al. (1995) classify the context into three categories, where you are (local context that includes the resources that are in the user's physical environment), who you are with (social context) and what resources (ICT) are nearby. Chen and Kotz (2000) and Henricksen et al. (2002) distinguish between passive and active context-aware systems. A static context describes immutable aspects of a ubiquitous system, such as B. a person's date of birth. A dynamic context refers to a user or environment context. These contexts can be highly variable in space and time, e.g. B.Temperature. Morsee et al. (2000) modeled context in terms of six main dimensions related to what, who, where, when, how it is accessed, and why it is useful. In Table 7.1, the context types are classified here in terms of environment type: physical, human, and ICT. Table 7.1 also relates this context classification to that of Morse et al. (2000). The human user context defines the user's tasks to achieve the objectives and defines the social and physical or environmental context.

It is important to distinguish several different types of user context awareness, e.g. B. knowledge of identified and unknown persons, user stereotypes, personal preferences, and user goals (see Table 7.1).

Context-aware modeling of systems

217

Table 7.1 A classification of the main context types by environment type of the UbiCom system and according to Morse et al. (2000) Feature

description

Environmental (physical) context What kind of physical environment or physical phenomena Contextual knowledge, such as knowledge of temperature, light intensity, chemical or biological concentration, etc. Where Spatial awareness or location awareness: Where contextual awareness can be exploited This can be in the current location or relative to a location in Reference to one or more locations, e.g. B. the current location in relation to a starting or destination point or a route. When Time Awareness: When context awareness is useful: now, later, and during any activity This can be defined in terms of absolute time or in terms of time relative to some other event or state. wireless and mobile connection User environment context Who User context awareness: who could benefit from a person's context awareness. The user context is divided into personal and social user contexts. Personal preferences1: p. For example, a referee at a sporting activity may prefer to blow the whistle for minor or major sporting infractions. Personal user contexts can be further subdivided into: personal identity context, e.g. B. a designated person who is an arbitrator; personal stereotypical context, p. B. the pattern of action in a sporting event indicates that the person is some kind of referee User activity or task Context: describes the user's current situation, e.g. B. Person Standing or Walking Social Context (User): Describes how someone's actions may affect others, e.g. for example, the whistle of a sports game or of a referee, police officer or spectator has different effects on others depending on their role in society Purpose Context for applications or users Why (task) User or purpose of the application: by what a context is useful for, the general application or user purpose for which the context is used, e.g. for example, a place is used to show someone or something in relation to its target Context matching: how the current context can transition into the target context Note: 1 Personalization is the user's awareness of a person's preferences (Section 5.7 .3).

Each individual context is defined by a metacontext, information that describes the context. For example, the location context must also define what type of location coordinate system and units are used. Also, the relationship of the current context to other contexts may be important, e.g. B. the relationship of one's home to the nearest medical facility. Note that the focus is actually on different contexts. The most important context invariant is the user and her characteristics, such as her identity and perhaps her contact details. The transition from the current context to the destination or target context follows a fixed or dynamic mapping. This represents a pre-planned form of adaptation to the context (Section 7.2.3). Context often implies a situation that affects a system external to the system, the environmental conditions surrounding the system (external context), which Morse et al. (2000). Equally relevant are the internal system conditions and the known terms of use of a system (internal context). More generally, we can refer to a context as something that relates to or influences the use of the system, i. h either through internal or through

Context Aware Systems

gle

without to gle rib

in the new l

is the time

Over there

Tick

are you

inis

generation

expression

Mes

stochastic

Von

le ltip mu ribute that

semi-automatic attic tom

T IC Mensch

Physically

IPL

sin

Unless

lat

lieutenant mu

With respect to

finally

To adjust

Bury

external to

218

Figure 7.1 Multidimensional multilevel support for a UbiCom property, e.g. B. Knowledge of the context

external interaction The multidimensional characteristics of modelable context types are summarized in Figure 7.1.

7.2.2 Context generation and context composition New contexts can be created in real time using sensors located in the physical environment, e.g. B. Temperature sensors. Often, the lower-level raw contexts emitted by sensors must be further processed into higher-level contexts that are relevant to users and applications. The collected values may need to be scaled or transformed in different value ranges or domains, e.g. B. an electrical signal from a temperature sensor is associated with a temperature value on a Celsius or Fahrenheit temperature scale. The captured values may also need to be related to other entities to be useful, e.g. For example, absolute position coordinates can be associated with a position context such as a particular building, postal code, or other country area identifier. Rather than use the location coordinate as a low-level context, it often makes more sense to access an abstraction of the context, e.g. B. "This photo was taken at these people's homes." Some contexts, such as location, entity, activity, and time, act as sources of contextual information from which other contexts can be derived: derived contexts or context reuse. Combining multiple individual context values can lead to a more accurate understanding of the current situation than considering an individual context.6 This is particularly useful in iHCI system applications, as it can avoid users having to explicitly identify themselves. themselves or their tasks and objectives. For example, knowing the current location and time, along with the user's calendar, can allow an application to infer user context, such as the user's location and time. For example, having a meeting, sitting in a class, waiting at the airport, etc. User contexts include:

• Combination of multiple simple contexts (context composition). Systems may need to handle mediation and interoperability between multiple contexts. The order in which the contexts are combined can be important. Finding and matching contexts cannot scale as uncertainty and complexity increase.

The combination of contexts can also generate conflicts (Section 7.2.7). There are several challenges with this simple calendar context approach, e.g. for example, users may have registered multiple appointments and not register which appointment has priority.

Context-aware system modeling*

219

Combine homogeneous contexts: e.g. B. from multiple independent sensors due to variations in individual measurements. Mixing of heterogeneous contexts: This can be used to determine a composite or main context, e.g. B. User, context. Inferring high-level context from low-level context: For example, knowing a person's weight and approximate location could infer a user's identity. Deriving a lower-level context from a higher-level context: A location can be determined from annotated locations, such as a street name and building name, and passerby information.

• Query a user profile or the user settings, p. B. the calendar of a user to know who the user is

• •

do at a certain time. A user profile or calendar may not contain all the relevant information about activities. Users can also spontaneously stray from your calendar or profile. Contexts may not be fully deterministic or fully observable. Ask users (Section 5.6) to define their user preferences: Closed queries (choose from a set that best describes the context) can be used versus open queries. User observation (chap. 5.6), p. B. Using computer vision based on camera technology and image processing, identify features such as faces, fingerprints, etc. and then bind them to the user context.

Context-aware systems may need to support contextual annotation maintenance, including creation, modification, and deletion. A challenge here would be to harmonize or standardize the annotations so that all users can use them consistently. Security, especially access control, could be useful in certain apps to protect privacy or restrict access. For example, users may want to share their location with friends but not with the general public. Context-aware systems may need to be able to link heterogeneous contexts. Contexts can be of the same type but represented differently and defined using different meta-contexts. For example, more than 100 different location coordinate systems are used; the same type of context can be annotated differently. The same context can have different semantics depending on the application that uses it. For example, a person who stops in front of a building may be waiting for someone or something, or simply interested in looking outside. A composite context may be necessary to interpret the meaning and usage of an individual context, e.g. B. in this case to determine not only the location of a person, but also his gaze. Stopping and looking at the building can indicate an interest in architecture, while stopping and looking away can indicate that someone is waiting for something more.

7.2.3 Context-aware adaptation The current context often appears to be the main focus in a context-aware system, but in reality it is the relationship of the current context to a target context that is the essence of context awareness and adaptation. . Of all the possible contexts that can be determined or are available, only the current contexts that belong to the target context of the particular application are important. Running an action plan moves the current context to the target context. There are often limitations on the target context and how much the actual current context can vary from the planned current context. Although the value of the current context changes and different parts of the same general plan are executed depending on the context, the general planned behavior of the system often remains constant, it is not adaptive (Figure 7.2). There are several issues that make context matching more complex in practice. First, in order to achieve the target state, the action plan is sometimes conditional and may fail in various ways: the plan may be wrong, the actions may fail, the state of the system may have changed, etc. Therefore, the execution of the conditional plan is required to capture the deviations in the plan, which can be recognized by

Context Aware Systems

220

start

advance

move to the side

Planned current context

context deviation

Replan and move forward Objective context

Planned current context

Move context offset to page

Figure 7.2 A context-aware conditional scheduling model based on preplanned actions that move the system toward a target context

Capture the actual context and compare it with the planned context. Scheduling is discussed in more detail in Section 8.7. When a context mismatch occurs, the planned behavior of the system must be modified to meet the target context, this is known as context-aware adaptation. Second, context awareness may require the support of multiple independent or interdependent target contexts, e.g. B. Collect income by maximizing the number of goods transported and punctuality. In some applications, eg. B. In the Personal Memories scenario, there are multiple user target contexts: first, to capture and record people, and second, to annotate the record to improve later retrieval of the record using the contexts of where, when, and who is at the recording. . In both cases, inherent to this application is the ability to associate a recording with the combined current location, time, and people identified in the recording context for the recording. The location and time part of the current context can be automatically generated by time and location sensors, respectively. If a goal of the context-sensitive personal reminder app is also to capture good quality images of people (target context), a combined face detection and face autofocus system is required to capture images of people in photo capture. context (current context) into one in which the camera automatically focuses on the detected faces of the people in the frame. The context match of the current context with the target context may be limited. The context constraints in the Personal Memories scenario are that for autofocus, subjects must be in front of the camera and at close range and recognize the subject's context match to the work. Sometimes context constraints apply to the target context, e.g. B. only focus on the people in the AV recording and sometimes the contextual limitations of the setting apply, e.g. B. Only use major roads to move goods or people from the starting or current location to the destination location. Sometimes there can be a contextual single shot or single step match, e.g. B. Calculate a preset route from the start location to the destination location and sometimes multi-shot context matching can be used, such as: B. in an automated vehicle collision avoidance system. The physical world context can itself describe associations of interest in addition to the relationship of the current context to the target contexts. The relative location, e.g. B. where someone lives or where a sporting activity took place may be of more interest than the absolute location. Sometimes absolute time is recorded. At other times, the relative time is of more interest, e.g. B. how old someone was when the picture was taken. These spatial and temporal relationships can be specified at the time of recording or generated at a later time. In an active context-aware system, the system performs context adaptation, but in a passive context-aware system, the system presents the context to human users and they must perform each context adaptation manually. Context-aware users and applications may require adaptation to a composite target context, such as B. a combination of place, person, time, and ICT access system. Design questions are concerned with how

Context-aware modeling of systems

221

form independent and dependent individual contexts, e.g. For example, a late bus tries to minimize its delay by not carrying more passengers on the route, even though this would generate more revenue for the bus company. Sometimes a context customization workflow is needed to sequentially serialize the customization of multiple individual context customizations. The order in which each adjustment is made can be important, e.g. B. Determine the location and adapt to the location, then determine the user preferences and adapt to the user preferences or vice versa, as this can greatly affect performance, i.e. search space size and complexity of the mediation of the context. An example of the problems with combining multiple independent local contexts is the use of a location-based meeting service that can suggest multiple common meeting locations based on the participants' current GPS locations and their route to the destination meeting point. Determining a proposed common context for a meeting can be complex, as multiple different weights of multiple parameters may need to be used to reach agreement, for example when travelers have different modes of transport, different current locations, different routes and different preferences for the type of meeting meeting points (Meng and Poslad, 2008).

7.2.4 Environment modeling The simplest type of context-aware system operates in an environment that it can fully observe and determine. It generally operates in an episodic environment that depends on the current state of the environment relative to the target state, but not on the past state of the environment. More sophisticated context-aware systems can also store each or a few of the current contexts, which then become the historical context, to represent, explain, and justify how the current or past goal contexts were actually achieved (Section 8.2 .2).

7.2.5 Context representation Strang and Linnhoff-Popien (2004) examined different representation types for context models and identified six different representation types. These were discussed and evaluated in terms of how well they support a distributed context model and composition; partial validation (relative to a formal schema for the context model); richness and quality of the information (the sense of the heterogeneity of the sensor and the temporal variation of the environmental context); incompleteness and ambiguity of the registered context; Level of formality (to accurately describe and share the importance of context) and applicability to existing environments (especially to web service-oriented frameworks). This is summarized in Table 7.2. Strang and Linnhoff-Popien conclude that of the six context representations, an ontology is the most promising representation for context models. However, in practice, this analysis is more oriented towards a web-like service-oriented infrastructure than towards a highly distributed, heterogeneous, scalable and low-resource embedded systems infrastructure that must be designed to handle ontologies. Some ontological representations appear to be closely related to a particular type of logical reasoning that may not be easily applied to support temporal, uncertain, and spatial reasoning. There are several practical problems with using an ontology-based context model for mobile users. Thin client devices and other resource-poor compute nodes acquiring contexts still lack the processing power to handle parsing ontology instances. Also, some context sensitive applications such as B. Location awareness, ability to handle large complex spatial data structures. The level of granularity at which semantic relationships are discovered and manipulated needs careful thought. It can quickly become difficult to think online about the detailed semantics of large data structures or large sets of contextual data measures. For context-aware scalable systems, it may be better to treat much of the data at a syntactic rather than a semantic level, using simpler handling of events, conditions, and actions at the syntactic level.

Context Aware Systems

222

Table 7.2 Different types of context representation according to the model of Strang and Linnhoff-Popien (2004).

Type of structure and how it is recovered

Observations

key value

Simple, flat data structure for modeling contextual information Get By: Linear Search

Pros: Easy to manage and analyze in embedded systems. Disadvantages: uses exact matches, lacks meaningful structuring, lacks efficient context retrieval algorithms, has a weak formalism, and handles incompleteness. May need multiple values

marking scheme

Hierarchical data structure, e.g. B. XML, consisting of user-defined markup tags with arbitrarily nested attributes Get By: Markup Query Language

Advantages: distributed model, uses underlying resource identifier and namespace model; XML web services are becoming ubiquitous; Dealing with heterogeneity, dealing with incompleteness Disadvantages: expressive structuring and weak formalism

Graphic

Graphical data structures and richer data types, e.g. B. UML (Unified Modeling Language), ORM (Object-Role Modeling) Obtained by: Transformation algorithms

Pros: More meaningful than key values and hierarchies. Disadvantages: support for distributed context model, handling of incomplete data, lack of formalism for automated online access

Object Oriented (o-o)

Context processing is encapsulated and hidden from other components. Access is only through certain interfaces Get by: Algorithms

Pros: distributed o-o is mature, some validation but often not very formal. Reuse can be supported through inheritance and composition. Disadvantages: Handle the incomplete

logic based

A logic defines the conditions under which a conclusive expression or fact can be derived from a set of other expressions or facts.

Advantages: strong formalism, expressive structuring Disadvantages: handling of uncertainty, time-varying instances, heterogeneity, often difficult to validate partially - or all or nothing, simple structuring, handling of incompleteness

strong ontology

An expressive conceptual model combined with a Get by logic: Reasoning

Advantages: meaningful structuring, handling heterogeneity, partial validation Disadvantages: handling uncertainty, scalability when searching large amounts of data, use in resource-poor integrated environments

7.2.6 A basic architecture La Baldauf et al. (2007) are also present in the high-level architectural model of a general context-aware system given in the basic context-aware model8 (Figure 7.3). It consists of four main components: capturing the current context, objective

To simplify the model, it does not support self-awareness and focuses on episodic environments where the system only needs to know the current context.

Context-aware modeling of systems

223

View user tasks to create user contexts

context processing

objective context

application

Environment (Env) Context Creation Abstraction Context Filter Configuration Sensors Events Environment

Fit Target–Env. context mediation

guidelines

annotation

custom context

Non-personalized content with context annotations

context composition

Save/Retrieve

context check

Discovery Access Control

Context storage Storage context detection

Actor access control

context management

Figure 7.3: A general architecture for context-aware systems

Context creation, adaptation of the current context to the target context and context management. The core design features of this architecture model are as follows. It is self-contained and can be reified in a variety of architectural styles (Section 3.2). It articulates in detail the distribution and interaction of four main groups of context-based functions that form a life cycle process for the operation of a context-aware system. It is also independent of how these different functions can be divided and distributed. Separate the concerns of using the context from the context itself. The context-aware operational lifecycle consists of: (1) the context of the physical, human, and ICT environment; (2) the data is created; (3) the user's target context requests are created; (4) environment contexts are used to customize some user or target context of the application; and (5) context data must be handled. This life cycle is discussed in more detail below. 1. Context determination Acquisition of environmental context: Sensors convert the (current) state of the environment, such as time, location, and physical phenomena such as temperature, light, sound, etc., into data for calculation. This can include many sub-processes, such as calibration and configuration of the sensor, e.g. B. to select the frequency with which measurements are taken. a. Capture of user contexts, such as the identity of the human user, stereotypes and personal preferences: These can be obtained directly or indirectly from the user's interaction. b. Encapsulation and abstraction (post-processing of raw contexts): a context in a service: encapsulation to allow access to the context through a published interface; Context data abstraction by interpreters, e.g. B. harmonize heterogeneous context values into a common representation; Convert between different representations and structures for the same type of context.

Context Aware Systems

224

C. Filtering: the context of the environment in relation to the context of the user. Filters only consider events within a specific scope that match the defined contextual policies or restrictions. 2. User Context Collection a. Recording of the target user's context: This can be derived from the users' application tasks. b. Policy or Constraint Capture: Created from user tasks to determine how environment contexts mediate user contexts. C. Encapsulation and abstraction of the user's target contexts. 3. Context processing a. Context Composition: An application can control the use of multiple environmental contexts, which can be captured from multiple context sources. One context type can be used to affect another context type based on a user context policy. b. Mediation: Various environmental contexts are linked and related to each other. C. Adaptation: passive or active or control: i. Passive (or Presentation): In a passive context-aware system, the surrounding context is used as a constraint to select or query information from an application or user context and simply present it to the user. The user controls an application manually or adapts its actions to the context. ii. Active (or Automatic): Adaptation to the context of the application or the user automatically adapts to the surrounding context. It can be of the push or pull type. iii. Control: The user context can be used to control the environment. 4. Context management a. Discovery: Directory services enable the registration and discovery of context sources, stores, and users. b. Storage: Of contextual data in a data resource, storage may include history-based organization of the stored context and support for fast query-based retrieval. C. Sharing the target environment and contexts so they can be distributed and made accessible. namely. Access control: Protects the privacy of all contextual information that can be linked to human owners. In this architecture, the mediation or adaptation of the application or user context to the environment context is performed in a subsystem (middleware) separate from the application itself, so that the mediation elements can be reused between applications. Conversely, a context-aware system can also be designed so that customization is part of the application because the application has specific customization requirements, rather than being outsourced to middleware. Different processes and algorithms may need to be used for context detection. Determining the environmental context may require information from multiple sensors, e.g. B. for positioning, they are added. Determining a user context may require the aggregation and composition of multiple contexts from multiple sources in potentially multiple application domains at multiple levels of abstraction. uncertainty justification, p. for example, there is only the probability that it will rain and arrangements need to be changed at an outdoor event and conflict resolution e.g. B. The journal indirectly says that someone must be present in a room, but a door entry authentication system has indicated that no one has entered the room. This may be necessary when combining multiple heterogeneous contexts. Matching a user context with an environment context may require search algorithms (Section 8.7.1) to search a very large context data space to find specific matches. Context matching may also require complex (semantic) metadata models (Section 8.4) and mediation processes (Section 9.2.2) capable of matching in heterogeneous context spaces.

Context-aware modeling of systems

225

Further specification details and examples of these processes and algorithms are provided in relation to the context awareness subtypes: mobility awareness, spatial awareness (everywhere), temporal awareness (anytime), human user awareness, and awareness. of ICT.

7.2.7 Challenges in the perception of context Henricksen et al. identified some of the main challenges in context modelling. (2002), Strang and Linnhoff-Popien (2004), Table 7.3 expands on this theorem. Of the challenges in Table 7.3, challenges 2 and 6 are highlighted. The combination of automatic context matching with incorrect, incomplete, or imprecise context determination can lead to many false positives and false negatives for user contexts, which which leads to unsatisfactory user experiences with users who prefer a manual system without automatic context matching. Some possible solutions are listed in Table 7.3, some of which will be quite difficult to implement. For example, the system could provide feedback on why a particular user context is likely to persist, but this dialog can distract users from their tasks. In addition, the inner workings of the algorithms may need to be revealed, but these may require specialist knowledge to understand. Also, the large amount of contextual information that can be collected, i. h Challenge 5 Being expensive to manage and containing a lot of data of very little importance. Table 7.3 Main challenges in modeling contexts Challenge

causes

possible solutions

1. User contexts can be incorrect, incomplete, imprecise or imprecise, ambiguous

Implicit observations of user contexts may be incorrect, incomplete, or inaccurate

Use context composition to improve context accuracy

Users can provide wrong information if specifically requested

Use iterative processes of interaction with the user; Please provide further explanation. The most accurate user contexts take time to establish. Use machine learning and simulation to better determine and predict user context over time

Modeling user contexts from too little input data over too short a period of time Users of the context may imprecisely define their needs, e.g. for example, find X that is close to Y 2 . Environmental contexts can be incorrect, incomplete, imprecisely defined, determined, or predicted.

Delays can occur when dynamic context information is exchanged. Interrupts or errors can mean that the path between the context producer and the consumer is broken, i. h part or all of the context is unknown

Use context composition to improve context accuracy. Manage context uncertainty. Partial contexts may need reasoning to predict the part that is not easily accessible. Further context justifications may be required to define a context.

3. Contexts can have a series of spatiotemporal characteristics: local effects in time and space

Some types of contexts are immutable, while other types vary greatly in time. Context generators can be mobile or vary between regions (spatial variation).

Define how often to capture each context based on its variability. Define interpolation and aggregation to combine values. Handle context unpredictability

(continues overleaf)

Context Aware Systems

226

Table 7.3 (continued) Challenge

causes

possible solutions

4. Contexts can have alternative representations

Sensor data post-processing level Multiple representations exist for the same context in different forms, at different levels of abstraction and perspectives, depending on usage

Capable of supporting and mediating between multiple context representations

5. Contexts can be distributed, partitioned, and made up of multiple parts that are highly interconnected

Contextual relationships can be obvious or less obvious. They may be linked by derivation rules that make a context dependent on other information (context). Composite contexts may need partial validation because not all parts are always accessible

Use rich conceptualization to provide context

Large and complex environments can be explored, many sensors can be used Focus is on archiving context data instead of applying context Many contexts may be available but users are not sure which ones make sense to apply

Filter context information before saving

6. Context awareness can generate huge amounts of data

Many remote events can replace fewer local events

Partial contexts may need to be considered to predict the part that is not easily accessible

Use appropriate search and archiving techniques for storage. Use data mining techniques to analyze data. Use active associations to discover connections; otherwise they will remain hidden

7. Local sources of context and processes may need to be embedded in resource-poor infrastructures

Context sources, such as sensors, are highly distributed, mass-produced, and integrated into inexpensive, low-cost, and resource-poor embedded systems.

Consider the trade-off between the expressiveness of context messages and the resources required to process those messages.

8. The use of context can limit people's privacy

Contexts are often inherently tied to people in order to be useful.

Information security is necessary to protect contextual information.

9. Awareness of context cues and changes can overload or distract users

Autonomy and knowledge of the status of many individual system components and the need to maintain and update. When context switching is automatic, user control is reduced and use may be interrupted

Make sure context moves happen automatically and don't interrupt users

Challenge 6 highlights the deluge of data and metadata that context-aware systems can generate. One useful technique appears to be to support the ability to set conditionally triggered contexts or future context awareness, but keep them hidden from users and applications using context-aware middleware services until the conditions that trigger them become true. Mann gave an example of this in Section 2.2.4.5. Where a specific user's shopping list context is triggered only when a user is detected to be near or entering a specific store. The condition

mobility awareness

227

The concept of activated context also allows contexts to be established by one process and consumed by another process, and for context establishment and activation to be separated in time (delay) and space.

7.3 Mobility Awareness 7.3.1 Call Routing for Mobile Users One of the earliest examples of context-aware applications is the Active Badge Location System by Want et al. (1992), started in 1989. This application was intended to be an aid to a telephone receptionist, before cell phone networks became widespread, so that employees could be contacted when they were not at their desk or at home. Once a person was located, phone calls could be routed to a desk phone closest to the person's location. The system uses knowledge of the user's location to route calls to the nearest landline. Sensors have been placed in offices, public areas, and main hallways to detect the signs of worn active badges; Of course, some private areas were also defined in which people could not be monitored. The system provided a table of names against a dynamic update field containing the nearest phone extension and a description of that location – an early example of digital call routing for mobile users.9 Want et al. (1992) did not specify the more general concepts of context and context-aware, as they focused on location determination. Today, two decades later, location awareness is routinely used by many mobile applications to provide additional types of location-based services.

7.3.2 Cell Phone Location Basic cell phone location involves determining which cell phone transmitter and its service area (your cell) is closest to a phone. Mobile phone users are typically registered in a Home Location Register (HLR) database by a Mobile Switching Center (MSC) maintained by a mobile network operator. A visitor location register (VLR) database is also maintained in an MSC for a group of mobile phone cells within a location area (Figure 7.4). As users move between areas, a cell notifies its VLR that the user is entering or leaving its location area. When a call is made from User B to User A, the call first queries User B's VLR to see if it knows User A's location. If it is not there, a call is made to User A's HLR since each VLR that Visitor A notifies A's HLR. A's HLR tells B's VLR which VLR A is on. A type of query called paging can be used to locate the particular cell where user A is located (Pashtan, 2005). Emergency call statistics show the increasing prevalence of calls made by mobile phone users. Unlike a fixed location phone, however, mobile phone users can sometimes provide inaccurate location information. In the late 1990s, a US government order initiated US E911 programs to establish a minimum location accuracy for mobile phone users in emergency situations. Similarly, an equivalent mandate was initiated in the European Union in the early 2000s, EU E112 (Pashtan, 2005). While cellular networks routinely need to determine the location of the phone within a cell10 to route calls (Figure 7.4), enhancements are needed to the cellular phones (hardware and software) or to the network, or both, to support a determination of more precise location within a cell. . GSM mobile network operators may use

9 This first context-aware mobile application is essentially a type of local digital call routing for cell phones or cell phones. Although global cell phone systems became available in the mid-1980s, they were either analog or 1G systems. Digital or 2G cell phones only became available in the early 1990s (Chapter 11). 10 cells may be around 150 meters or less in urban areas, but around 30 kilometers in rural areas. TDOA, GPS, and hybrid techniques can improve this accuracy to 30-50m, but GPS doesn't work indoors.

Context Aware Systems

228

Group of 1 or more cells User B calls User A

Cell/Base Station Visitor Location Record

User A has moved

MSC-Benutzer A

mobile exchange

Benutzer B. 1999

VLR

HLR

Home Location Register Packet-Switched Core Network

Benutzer C

MSC Figure 7.4 Location determination in mobile networks

three different enhanced positioning methods based on enhanced observed time difference of arrival (TDOA) between the phone and multiple base stations, GPS (phone compatible, for outdoor use), and hybrid techniques such as A-GPS (assisted GPS) that combine GPS with information from the network.

7.3.3 Mobile user awareness as an example of composite context awareness Mobile context awareness is an example of composite context adaptation. First, spatial awareness is used to adjust activities relative to your location. Spatial awareness may not require access to remote data services other than position determination, since position could only be used to select locally stored static data, e.g. B. Maps used in navigation systems, see Section 7.4. Location tracking is becoming commonplace in modern digital phones, cameras, and other recording devices, allowing users to automatically note not only when, but also where photos were taken. Information retrieval from remote sources can be customized based on user preferences. ICT context awareness (Section 7.5) is useful for mobile users as it tailors content being accessed remotely to better match the characteristics of mobile access devices and improves available bandwidth in the local wireless access loop. Content adjustments can also be made regarding personal preferences.

7.3.4 Tourism services for mobile users The CRUMPET system, Creating Personalized User-Friendly Mobile Services for Tourism (Poslad et al., 2001) is an example of a context-aware composite application. In this system, tourist information services such as maps, routes and recommendations for places of interest can be adapted to a spatial context that affects the current location, the personal context of a service, the network context and the terminal context, see Figure 7.5. The context sensitive architecture of the CRUMPET system is based on a multi-agent distributed system design. Typical user interaction (context of use) for a given service triggers composition

spatial awareness

Map components: • Nearby “world” map • Start/edit tour • Status bar with proactive “light bulb”

229

My IP address and port are...

Ok ok, here are your local attractions. Interests.

Hereismy here my new location. new location.

Components: • Map of the “world” • Diagnostic information • Client status (agent and network status) • Points of interest

Figure 7.5: A composite context-aware application (site, person, terminal, and network)

Context customization (network, device, location, and user) (Figure 7.5). In the CRUMPET system, a specific order of context-aware personalization is used as follows. The storage and display capacity profile of a user's access terminal is exchanged with the system during the initiation of a user session. For example, localization is used twice (Poslad et al., 2001). First, a user's current location can be used to define a user's query and further filter relevant information. Unless the relevant location is explicitly indicated, the user will receive information relevant to their current spatial context. Second, a user's movements within a region can indicate the region's interests. For example, if a user visits several old churches, they are likely to be interested in the churches and perhaps also other historical buildings in that city, such as the old town hall. Users generate many potentially interesting events as they move. This can be leveraged for user modeling and to recognize and anticipate relevant user interests. Therefore, the context of the combined location and person model can be used to create a map of things of interest in a location. This is an example of environmental context composition where one context type (location) can be used to determine another context type (personal preferences) based on a user context policy. Finally, based on monitoring the performance of the mobile terminal local to the access node, the network profile, the content, e.g. B. a personalized location-based map, tailored to your device or network profile.

7.4 Spatial Awareness According to many, location or spatial awareness11 is one of the main drivers of mobile services (Marcussen, 2001). This allows access to services remotely from anywhere, that is, extending the

Spatial assets are understood more as a general and comprehensive term than as location-aware. Whereas spatial cognition is aware of a specific 3D space and possibly its structure, location cognition focuses on a 1D point in that space.

Context Aware Systems

230

Table 7.4 Some types of SAS applications with illustrative examples Type of SAS application

examples

Navigation context switching Location context query Personal emergency Business asset tracking

Where is the nearest gas station? How do I get to X? Long queue of traffic ahead, then turn right onto the A1 to avoid this. What is the speed limit on this road here? When does this building open? Medicine: I'm having a heart attack! On the road: My car has broken down! Why does delivery to this customer always take twice as long? Why is our van deviating from the standard route? Transportation: View estimated arrival times for trains or buses based on your current location. Please let me know if my child gets lost in the neighborhood. Where did Smudge the cat go this time? My car was stolen, where is he? Free calls to your mobile phone while you are at location X; Food specials etc. Users can publicly and privately share and sync their context (location, activity, mood) with friends; Organize an ad hoc meeting nearby

Public asset tracking Personal asset tracking Location/time based offers Location and time synchronization

experience mobility. Spatial Reconnaissance Systems (SAS) are commonly known as Location Awareness Systems (LAS), but the term Spatial Reconnaissance is a more precise term, as this type of system is often not only aware of locations, but may also need spatial knowledge. features ranging from simple point location features to polygon-like two-dimensional areas and more complex irregular three-dimensional spaces that change over time and their spatial relationships. A passive SAS type presents only a relative or absolute location context to the user. A more active SAS automatically adapts the application output to the context of the location, e.g. For example, a "SatNav" application, which uses a satellite-based global positioning system, can display up-to-date current positions and routes on maps. A more active system may try to use the feedback to control a local environment, e.g. B. to change the human population density in an area over time, e.g. B. by providing attractive business offers specific to an area. There are many examples of SAS applications, see Table 7.4. A typical spatial system consists of spatial (environmental) context determination, user context determination, spatially based adaptation of user context, and management of spatial context data stored in a GIS or geographic information system. .

7.4.1 Spatial context creation 7.4.1.1 Spatial capture There are a variety of methods for capturing structured or vector spatial data in GIS. A digitizer can produce vector data while an operator draws points, lines, and polygon boundaries on a map. A scanner produces a raster map that could be further processed, e.g. B. use edge detection to generate vector data. Survey data can be entered directly from digital data collection systems; Surveying instruments or other remote sensing, such as satellite remote sensing, provide another important source of spatial data. Satellites can use different types of sensors to passively measure the reflectance of parts of the electromagnetic spectrum or of radio waves emitted by an active sensor such as radar. Remote sensing collects raster data that can be further processed to identify objects and classes of interest, such as land cover. Analog aerial photographs can be digitized and features extracted. Stereo pairs of digital photographs allow data to be

spatial awareness

231

recorded in two and three dimensions, measuring heights directly from a stereo pair using principles of photogrammetry. As high-quality digital cameras become cheaper, they can be used to capture spatial objects directly. Location systems can also be used to determine location coordinates. However, these must be used in combination with annotations, also called geoattributes or metadata, which are linked to location coordinates.

7.4.1.2 Location detection There are several basic location detection methods based on triangulation, lateration, scene analysis, and proximity (Hightower and Borriello, 2001). Triangulation is one of the most universal positioning techniques. It uses the geometric properties of triangles to calculate the positions of objects as a function of distance (lateration) and angles or orientation (angulation), see Figure 7.6. Lateralization determines the position of point O with respect to three reference points A, B and C. RA measurement determines the point on circle A, RB measurement determines the points where each point is located, at the intersection of the circle A and circle B. By measuring a third point, it is determined to be location O. An angulation algorithm can be used to determine the position coordinates of location O by determining the line-of-sight angles at two points A and B and knowing the distance between A and B. Lateralization and angulation are often combined in meaningful ways to know not only where an object is, but also what direction it is viewed from. Lateration uses two main techniques to determine distance in practice as a function of attenuation and time of flight (TOF), also known as time of arrival (TOA). Attenuation, also known as Received Signal Strength (RSS), estimates the

X Y RA A O

RB B a

Lateration

angulation b

CDN

3 equations to determine the location of point O w.r.t. known locations A, B, and C on a 2D plane RA2 = X2 + Y2 RB2 = (X - (AO + OB))2 + Y2 RC2 = (X - AO)2 + (Y - OC)2 Use substitution to get X and Y X = (RA2 - RB2 + (AO + OB)2)/2 (AO + OB) Y = (RA2 - RC2 + AO2 + OC2)/2OC) - AOX/OC

Knowing the distance AB, the angles at A and B, X and Y can be determined using simple trigonometry Sin A = Y/a Sin B = Y/b Y = a * Sin A = b * Sin B Cos A = X/a X = a * cos A = AB – b * cos B

Figure 7.6 Using lateration to determine the position of point O with respect to three reference points A, B, and C (left). Using angles to determine the position of O with respect to two angles for the line of sight from two points A and B and knowing the distance between A and B

232

Context Aware Systems

RF signal strength at the receiver, which knows the signal attenuation as a function of distance and the strength of the signal transmission, e.g. B. 1/d2 for the simple case in free space, where d is the distance between emitter and receiver. In practice, however, distance and time measurements are complicated by variable attenuation due to humidity in the air, etc., multipath effects, reflections, point-to-point interference, and the fact that time transmission is not precisely known. An example of RSS technique is the use of WLAN mitigation measurements (Saha et al., 2003), which can be performed on mobile devices with WLAN access. They claim an accuracy of 1 to 7 meters, depending on the signal processing used and the number and distribution of WiFi access nodes or transmitters. In spaces such as indoors and certain outdoor areas where there are many obstacles that can cause reflections, refractions, and multipath effects, the attenuation is uneven, making it a less accurate method than TOA. However, some TOA systems such as GPS may not always be accessible as they require an unobstructed path between transmitters and receivers, see below. An example of a TOA system is a satellite-based Global Positioning System, or GPS. TOA measures the time the signal is sent versus the time it is received. The distance d between the transmitter and the receiver can be estimated if the propagation of the signal is known (distance d = time * velocity of signal propagation). This assumes accurate time synchronization and that the sender knows the transmission time, but this may not be practical. Measurement of the time difference of arrival (TDOA) at two or more receivers or sent by two or more transmitters can be used to provide improved estimates of the difference in distances between transmitters and receivers. Trilateration uses absolute measures of arrival times from three or more locations. It is a method of determining the relative positions of objects using the geometry of triangles, similar to triangulation. Unlike triangulation, which uses angle measurements (along with at least one known distance) to calculate the location of the object, triangulation uses the known locations of two or more landmarks and the measured distance between the object and each point. reference. To accurately and unambiguously determine the relative location of a point on a 2D plane using trilateration alone, at least three reference points are generally needed. Signals can be measured relative to multiple transmitters to correct for any variable attenuation. An example of a triangulation system is the VHF omnidirectional ranging (VOR) system used for commercial aircraft navigation. It uses fixed ground transmitter stations that transmit two distinct signals: an omnidirectional station identification signal and a rotary transmitter signal that is swept in 360 degree arcs so that the signal is in north phase and out of south phase. Using phase shift measurements from two or more VOR stations, aircraft can determine their position. VOR is complemented by GPS for aircraft navigation. Scene analysis refers to the use of abstractions or markers of a terrain scene that can be correlated with previously recognized and stored markers to infer location, e.g. for example, a vehicle can orient itself to features in a terrain view, such as a railroad track. Someone with a cell phone and a camera can take a photo and transmit it to a remote service for recognition. The advantage of scene analysis is that it does not require an active signal transmission network, which requires power and can compromise privacy, but instead relies on passive observations. Proximity scanning uses short-range transceivers at fixed locations to determine when appropriate networked objects, such as RFID tags (RF identifiers), come within range. The proximity LAS, by definition, determines the relative position instead of the absolute one in the simple case. The use of RFID systems for environment-based location of RFID-tagged assets, such as vehicles, people, and goods, is now widespread (Section 6.2.4). The choice of a location determination technique depends on several factors, such as location accuracy, range, availability, coverage, cost, and limitations (Hightower and Borriello 2001). Haza et al. (2004) also grouped location sensing devices in terms of accuracy of determination and degree of implementation in research laboratories, custom implementation, and

spatial awareness

233

Consumer Provisioning. GPS, Bluetooth, Infrared, and WiFi are the most widely used devices, but they have a relatively low accuracy of 10 to 100 meters. Devices with a short range of RF transmission, such as Bluetooth and infrared, are accurate to 1-100m and are often used to determine nearby location. A major problem with TOA and TDOA systems is that while they have a global range, they require line of sight with three or four transmitters depending on accuracy, likely limiting use to outdoor use where there are no high side obstructions. . Hybrid systems or assisted systems attempt to combine the strengths and minimize the weaknesses of multiple positioning systems. For example, a terrestrial cellular network could be combined with GPS in an Assisted Global Positioning System (A-GPS) and support improved accuracy of several meters or better outdoors and 20m accuracy indoors (Djuknic and Richton, 2001). or it can be combined with WiFi trilateration techniques. Van Greunen and Rabaey (2005) offer a comparison of scalability-based location detection algorithms in terms of number of network nodes and communication power.

7.4.2 Location and other spatial abstractions A location coordinate by itself is often not that useful, it is too low: it is the location relative to the location context that is useful and gives it its meaning. The location coordinate and its associated geographic area within a bounding box or region and any geographic assets of interest within that area, e.g. B. gas station, restaurant, are required. In forward tracking, the relationship between current coordination and target coordination, future route is useful, e.g. B. how far is the destination and how to get there. Of interest in backtracking is the relationship of current location timing to launch timing and past routes. It is also challenging to automate application-specific types of contexts, e.g. For example, a building can function as a restaurant, gym, conference room, etc. This requires an abstraction or a service like a geographic information system (GIS) to answer spatial queries like "Is there some kind of service X within 1 km of here?". GIS services represent real objects such as roads, land use, and elevation and link them to digitized geodata. Real world spatial objects can be discrete objects, such as buildings, or continuous fields, such as precipitation and elevation. The GIS data consists of the geometric object, e.g. B. point, line, polygon, etc.; geo-attributes or metadata such as ID, address in the form of postal code, postal addresses, mobile phone location, x, y, z, etc.) and associated feature types and attributes such as size and color.

7.4.3 User context creation and context-aware personalization 7.4.3.1 Cartography: adapting spatial views to different user contexts An important application of SAS is cartography (Clarke, 2001), the use of maps to display different views of the spatial relationships between regions and points. Maps can show a selection of restaurants in a region or a route between a location and a destination. There are different types of maps for pedestrian, road and transport use, and to show boundaries and land use (cadastral maps), e.g. B. to show potential locations for new retail space. Maps are a reduction and abstraction of the geographic physical space we inhabit that describe places, regions, and their attributes. Cartography or mapping applications require the use of specialized data management systems to store, manage, and search large spatial features in GIS systems (Section 7.4.4) and to process places and regions and associate them with attributes of interest (Section 7.4 .3.2) . Unlike raster or image maps, which are only rendered as one layer, vector maps can be stored as multiple layers, from simple spatial views to more complex features such as points, lines, etc.

234

Context Aware Systems

Segments that represent buildings, roads, and rivers to regions that contain collections of them. Maps can be dynamically generated to show only the plans and objects of interest to simplify spatial visualization for users. Software such as Geotools, an open source GIS toolkit (Garnet, 2007)) provides APIs for adding, removing, and manipulating layers to create a map that is retrieved from a GIS, e.g. B. using the Open Geospatial Consortium (OCG) Geographic Markup Language (GML). Maps may need to be adjusted to fit screen resources, e.g. a mobile device and to support downloading the map via a wireless connection (on demand) instead of a wired connection (pre-cache).

7.4.3.2 Geocoding: mapping location contexts to user contexts Algorithms such as those based on geocoding (Ratcliffe, 2001) can be used to associate a context of use with a spatial context. Geocoding maps spatial locations (x,y coordinates) to and from street and postal addresses. Individual address locations, which are points rather than segments, can be derived by interpolation or estimation by examining ranges of addresses along a street segment. The individual addresses can then be linked to other metadata, e.g. B. with the type of companies that operate at a particular address to support inquiries about an instance of a company. Other algorithms are used to help with address matching when the addresses differ in spelling. Address information about which a specific entity or organization has data, e.g. B. the post office, may not completely match the terms of the inquiry. There may be variations in the spelling of street names, community names, etc. Consequently, the user generally has the option of making the matching criteria more stringent or relaxing certain parameters. Semantic approaches could be used here to automate matching to a greater extent.

7.4.4 Spatial context query and management: GIS Different subsystem designs are required for the different main functions of a geospatial system: determining spatial context; searching the spatial context space and matching a spatial context with a user context; store spatial context and exchange spatial context information. Determining a higher-level query relationship, such as B. what is the next part of this route or where the gas stations are located in this region requires different types of algorithms than those used for location determination, mainly because the search space can be much larger and have many more spatial independencies that are evaluated Needs to be converted. Spatial data structures can be very large and complex structures that require sophisticated data organization and retrieval techniques to optimize the associated computation and reduce the time it takes to fulfill search requests. Spatial contexts are stored and retrieved from a GIS. Algorithms that support geospatial processing typically associate unlimited spatial structures, such as line segments representing roads, such as segments a–g in Figure 7.7, and spatial polygon structures with irregular boundaries representing areas or buildings, such as objects C. and D (Figure 7.7). organized as hierarchies in various tree data structures, such as B. Quad-trees (adjacent non-overlapping bounding boxes) and R-trees (overlapping bounding boxes) (Samet, 1990). A spatial query, p. B. Determining the path from object C to object D involves consulting the R-tree to determine in which bounding boxes the spatial objects of interest lie, and then identifying the intermediate bounding boxes that contain the path segments, e.g. B. c to i in figure 7.7. The Open Geospatial Consortium12 has proposed several conceptual ways to extend an SQL RDBMS to support the storage and query of geospatial data, including defining operators such as

OCG 1997, see http://www.opengeosapial.org/, accessed December 2008.

Temporal awareness: coordination and planning

B2B2

gram

235

Hola

E B6

B3 B4 B5

B6 B7

A b c C B bounding boxes are used to enclose and represent spatial structures

from

An A

hide

The set of B is indexed to an equivalent R-tree

Figure 7.7 Storing and indexing spatial structures in an R-tree to support efficient spatial queries

Longitude, number of points in a route, distance between features and right of. Many relational databases include spatial extents. Data usually needs to be edited after it has been entered, e.g. B. Vector data must be "topologically correct" before it can be used for advanced analysis; Errors such as undershoots and overshoots must also be eliminated. Also, there are many ways to specify spatial points and other structures, e.g. For example, there are over 100 different coordinate systems for positions. One of the reasons for the different coordinate systems is that there are different corrections for 2D projections of the Earth, which is not a perfect sphere in different regions, e.g. B. North American Datum 1983 (NAD83) works well in North America but not in Europe. Measured location coordinates (eg, from the GPS system) and coordinates of geospatial objects (eg, a building or street point) in GIS are likely to vary between regions and between a map representation . Recently collected location data may need to be converted before use.

7.5 Time awareness: coordination and planning Time is often captured as a central attribute in context-aware systems, it is inherent in human work and leisure activities. Freeman and Gelernter (2007), in the Lifestreams project, proposed that time-based streams should be used to manage personal information, rather than the static file name folders used in most data systems. information at present. Many activities require space-time awareness. They must be synchronized with a specific space and time, e.g. to school, university, work, shopping, theater, etc. Gardner (1983) considers an individual's complex spatiotemporal perception and problem solving when using their body Body: To perform a complex surgical procedure, a series of dance steps to execute, or to catch or hit a flying ball, such as an important form of human intelligence called bodily-kinesthetic intelligence. In the following sections, aspects of time awareness with respect to the formation, distribution, abstraction and adaptation of the user context to the temporal context are considered in more detail.

7.5.1 Clock Synchronization: Time Context Generation Clocks and embedded timing devices must be synchronized and re-synchronized so that different users, services, and processes share a common time context. Timers can be set manually as they are not network aware and there is no global absolute time. The time can be manually set using a proprietary local clock configuration interface by someone who frequently reads the time.

236

Context Aware Systems

another source of unauthorized surveillance: this leads to human error. Device timings must be reconfigured periodically as timing circuitry degrades, or because they do not keep exactly the same time over a medium to long period of time and drift around the clock chip due to temperature variations. Clocks and timers that are network-enabled can be automatically synchronized with external authoritative time sources, such as those based on Universal Time Coordinated (UTC) services. The Network Time Protocol or NTP13 (Mills, 2003) defines both an architecture and a set of protocols for time synchronization that maintain time with an accuracy of tens of nanoseconds. NTP uses the intersection algorithm to allow NTP clients to select between multiple time sources. Multiple time sources are needed in case one fails. The intersection algorithm is essentially a type of matching algorithm for estimating the exact time from a set of noisy time sources and then finding their intersection. NTP is used in conjunction with an organizational hierarchy of primary NTP servers associated with atomic clocks. The secondary servers are synchronized with the primary servers. NTP clients can configure their time to gradually or incrementally change the accepted UTC time when their time deviates. The latter method avoids major interruptions in a time-sensitive ongoing process. Van Greunen and Rabaey (2005) state that traditional time synchronization algorithms such as Mills' (2003) achieve maximum accuracy regardless of computational and communication costs, and therefore energy expended by the algorithms. They discuss some synchronization algorithms based on lightweight tree synchronization and algorithms used in sensor networks that compensate synchronization accuracy with less power consumption and communication.

7.5.2 Temporal models and abstractions Time can be modeled as a linear progression, pointing from the past to the present, through various parallel events and activities, the arrow of time, and then branching off to various future times. It is also known that time varies for different users depending on the relative speed difference between them, as defined in the theory of relativity. This effect is not significant for users moving well below the speed of light (Hawkins, 1988) and is therefore not considered here. Time can be modeled as a moment, a period that represents the set of instances between two defined points in time, and an interval or duration, a period of time that does not have an associated start or end instance. Immediate events or activities with a very short duration are defined as a moment, while activities are often defined in terms of a start time and interval. Temporal models are made more difficult in practice by the coexistence of several times. There are several times that vary in different geographic regions to harmonize the time in different geographic regions with natural daylight (astronomical time). Time zones that correspond to different geographic regions and are oriented in an easterly direction are standardized as offset times from a reference time such as UT1 or GMT (Greenwich Mean Time). This simplifies time tracking for mobile users who travel and users who trade between time zones. This is compared to earlier train travel in the 19th century, where people could travel on trains operated by different railway companies that had different schedules because their terminus stations or main offices were in different time zones, quite confusing for the traveler. There are different syntaxes for date and time stamps, e.g. for example, one could use a year-month-day-hour-minute versus day-month-year-hour-minute with or without time zone offset format, etc. Therefore, translators are needed to convert between different date and time formats.

NTP is the longest-running and continuously running distributed application on the Internet, having its beginnings in the early 1980s.

Temporal awareness: coordination and planning

237

7.5.3 Managing the temporal context and adapting to user contexts Given a set of tasks to perform (the user context), a set of resources to use, and a set of time constraints (the temporal context), the goal of Task scheduling is allocating time and resources to user tasks. Task scheduling is easiest when tasks are completely ordered, start at predefined anchor times, require predefined time intervals to complete, are non-preemptive so that they are not interrupted and resumed once started, and use reusable resources without resource constraints. resources or shared resources. Simple scheduling may involve deriving a custom schedule that is a subset of another schedule that is known a priori, e.g. A simple planning process starts with an initial request to identify tasks and their time constraints, which are known in advance, then assigns them to resources and time slots, and then runs the schedule. When subsets of tasks of a known a priori schedule are to be selected, time can be used as a one-dimensional index to store and together with preferences to form a composite constraint on task selection (Figure 7.8). The assignment of tasks to human and other resources can be abstracted to form a graph of task nodes. Paths between nodes can be constructed to represent the program. There is a partial arrangement of tasks around those that have established anchor times and difficult time constraints. Other approaches include viewing tasks as a cause-and-effect story. Task scheduling is most difficult when there are fluctuations and uncertainties in time constraints, when tasks can be moved forward, when there are task resources and task coordination constraints, when resources are consumables, and when the semantics of a task event occur. time varies.

A simple job scheduling algorithm is to partially order n tasks on a graph and search to find a path 1T1

2T1

1Q2 1Q3 Known deadlines, but flexible execution and deadline

Task 1 Task 2: Break to eat and drink

Task 3 trips

task 4 free time

Known Design Known Times Deadlines

0 1T 1T1

1T2

1T3

2T2

2T 1T2

2T2

3T2

2T1

1T3

1T4 5T

4T2

3T1

5T2

2T3

1T4

2T4

Figure 7.8 Simple task scheduling for non-preventive tasks with a priori known execution times, deadlines, and periods without resource constraints

238

Context Aware Systems

For example, retailers can distinguish different transaction times based on when an item is ordered, when an item is paid for, and when it is delivered. In general, a timestamp is stored as an attribute of any environment or user context; h The value of this environment variable is valid at this timestamp.

7.6 ICT system knowledge To support the vision of ubiquitous service delivery over heterogeneous networks and access through heterogeneous computers and terminals, service delivery is enhanced when it takes into account the characteristics of the network and user terminal receiver. Otherwise, content may be delivered that doesn't display well or not at all, or may take a long time to deliver. Service customization should normally be transparent to most users, as users may not set their system properties correctly. For example, communication services for mobile users involve the use of automatically configured data paths. However, for certain services, administration policies may restrict automatic personalization. For example, banking and secure messaging services may need more restrictions on relatively insecure networks.

7.6.1 Context-sensitive presentation and interaction in the UI A user interface or UI facilitates the presentation and input of information by human users, supports queries for information about specific tasks, including instructions for inputting information, and the presentation of the answer. Universal access to content includes access to content through a variety of interactive devices with different capabilities. These can vary in size, weight, and mobility. They may have different display characteristics, such as screen resolution, size, and color depth. You can use different forms of input, including different types of keyboards, pointing devices, voice input, and gestures (Section 5.2).

7.6.1.1 UI Context Capture The UI context can be defined in a UI device profile. There are several specifications for how the UI profile is rendered. W3C CC/PP has defined the Composite Capabilities/Preferences Profile: Structure and Vocabularies 2.0 (W3C CC/PP, 2006). This defines a client profile data format and a framework for incorporating specific features of the application and operating environment, including terminal hardware, terminal software, and the terminal web browser. CC/PP does not define how the profile is transmitted, nor does it specify what CC/PP attributes should be generated or recognized, or what context markup language is used. CC/PP is designed to be used as part of a larger application framework. CC/PP is represented in RDF/S and an XML serialization. An example of a CC/PP device profile is shown in Figure 7.8. Strang and Linnhoff-Popien (2004) discuss the expressiveness of CC/PP version 1 and some attempts to improve its expressiveness. Several standards groups are actively developing standards related to accessing web services on mobile devices.14

For example, see the W3C Mobile Web Initiative, http://www.w3.org/Mobile/, accessed 2008, which contains device profile specifications. It is not clear how backwards compatible these specifications are with CC/PP.

ict system awareness

239

7.6.1.2 Content personalization A process framework is required to personalize the content of user interface profiles. The main focus in the adaptation of presentations was mainly the adaptation of content designed for decimeter screens and for small centimeter screens. Adapt content designed for small screens for viewing on large screens, e.g. B. Projectors, is simpler and could include dithering and smoothing to calculate additional intermediate pixel values to smooth out jagged edges when pixel groups on the screen are enlarged. Content personalization has two dimensions: transformation of the representation of the created content into one used on the access device, personalization of the interaction, personalization to use a particular device display convention, and personalization of the content itself. Bend.

• Conversion of the created content representation to the one used in the access device, eg. a map.

•

The defined GML is converted to an image for display. Custom content can range from passive to active, as it contains small code scripts in an application-specific program that generates output. Much research and development has gone into developing web-based content, including content-based scripting languages, which can include wireless mobile devices and wired PC devices. Pashtan (2005) discusses the variety of content language representations used for mobile devices, derived from three bases: Portable Device Markup Language (HDML), Hypertext Markup Language (HTML), and Compact HTML (cHTML). Different mobile devices support different web browsers that implement support for specific web content languages. The continued evolution and variety of ubiquitous devices to be developed will likely require multiple content submissions. Interaction customization: Efforts to standardize content languages have largely focused on image and text layouts. The W3C has developed a multimodal interaction activity designed to “allow users to dynamically select the mode of interaction that is most appropriate for their current needs, including disabilities, while enabling developers to create an effective user interface for the user. provide the selected modes. Depending on the device, users can provide information through speech, handwriting, and keystrokes, and the output is presented through screens, pre-recorded and synthesized voice, audio, and tactile mechanisms such as cell phone vibrators and braille strips (W3C MMI, 2002). Content customization: e.g. B. to display a large map on a small screen. This often involves more than just scaling, as important detail can be lost when content is scaled down. A common approach when preserving detail in content that is too large to fit on a small screen is to split the content across multiple displays and support techniques for navigating between them, such as scrolling. B. window stacking or window scrolling. Text display may reduce full text to show only the title or an abstract on a mobile device. Adaptation to a specific presentation style: to adapt the layout and appearance of a page for a specific device profile, e.g. Place the menu on the right for a landscape desktop screen and on the top for a portrait screen. It may also be necessary to follow the navigation style for a specific device profile. The presentation style can be determined in part by the appearance of the presentation.

Adaptation to different heterogeneous terminals should preferably automatically adapt the content to the capabilities of the terminal. There are two main approaches to this:

• Lowest Common Denominator (LCD) approach: Creating content that can be used on a few device categories covering a large number of devices. Each device in the category supports a lowest common denominator profile for that category. For example, in 2005, J2ME currently defines two categories (configurations) of ubiquitous devices. A connected device configuration (CDC) supports always connected network devices and one limited connected device

240

•

Context Aware Systems

The (CLDC) configuration supports intermittently connected personal mobile devices. CDC targets devices with more than 2MB of RAM and ROM. CLDC's Mobile Information Device Profile (MIDP) is intended for personal devices with a screen size of 96.54 pixels, a display depth of 1 bit, a one- or two-handed keyboard, a screen input device touch, 128 KB non-volatile memory and 8 KB non-volatile memory Storage for persistent application data. The LCD approach sacrifices the richness of some devices for the limitations of others. This can lead to unintuitive use of some complex interactions to keep interactions simple. Transcoding content to suit specific types of access devices: This involves transforming content from one form to another through well-defined mapping functions. Here, the input and output capabilities of the terminal must be distributed to the services that need to access it. The content can then be statically or dynamically adapted to the capabilities of the terminal. Please note that if the terminal functions are misidentified, e.g. B. by the human user, non-intuitive and useless interactions and useless content presentations may occur. Transcoding can be static (content is prepared in advance for specific target devices) or dynamic (content is dynamically prepared for specific target devices).

The ease with which content can be transcoded depends on how well the structure of the content is annotated. Annotations can be used as inputs for automated rules to convert content. Unlike database content, HTML content does not contain standard content structure annotations. A typical approach is to convert content, including active content, to an intermediate canonical format, e.g. B. Java bytecode, .Net Common Language Runtime (CL) code, or on the W3C platform, and the language-independent Document Object Model (DOM) used to define the structure of an XML document. So, three different threads are involved in its display: manipulation, which involves filtering, sorting, and sorting the parts of a document that are displayed; Transformation of document elements and structures related to a specific user and viewing context to another format, e.g. B. using XSLT (Extensible Style Sheet Language Transformation); Representation of elements appropriate to a particular screen, such as B. using XSL, the extensible style sheet language. Pashtan (2005) describes some good examples of using XSLT and XSL to transform content for mobile devices. In an ideal world, content should be independent of the delivery context. An author creates content once and publishes it in a variety of ways on a variety of devices, including desktop and other mobile devices. Currently, it is still very difficult, if not impossible, to efficiently design text and image-based content and interactions that can be viewed on both desktop and mobile devices, not to mention the broader range of ubiquitous devices, interaction support. multimodal. .

7.6.2 Network-aware service adaptation User mobility requires some services to know the address of the user's terminal when the user roams the network to route data to the user's new location, see Section 7.3.2. Mobile users often find themselves in an environment where multiple data communication networks may be available. Due to the variety of different network types and network characteristics, Quality of Service (QoS) can change dramatically depending on the network the user is currently connected to. Users on the go may encounter fluctuations in network conditions, which can be quite dramatic at times. One of the strategies used is that the wireless connection is monitored and content can be set to lower fidelity if a lower bandwidth connection is detected. For example, the round trip time can be monitored and used as an indicator to optimize the transmitted images for the available bandwidth. Content can be customized using downgrading mechanisms to reduce the quality of the content. Context awareness could also try to opportunistically obtain additional resources to maintain the quality of the content exchanged (Couderc and Kermarrec, 1999).

ict system awareness

241

A service that is aware of the properties of the physical network is called underlay-network-aware (Chapter 11) and can adapt its content transmission to the properties of the physical network, e.g. B. your bandwidth. Raz et al. (2006) discussed the requirements for context-aware services at the network layer. The dominant IP-based network layer design currently in use supports a simple and unreliable packet forwarding mechanism in combination with more complex routing protocols and the end-to-end TCP transport protocol. This requires improvements to support more flexible context-aware QoS delivery. To support this type of customization, some form of network context must first be defined and captured; h the state of the local network must be extracted from the different elements of the network. Second, the local network states must be assembled to create a global or end-to-end state. Third, these types of service-aware network models must be supported. This may include sharing and accessing network status information via open network APIs. Finally, active mesh adaptation models, in contrast to passive mesh adaptation models, can be designed to allow services to reconfigure networks to better justify their needs. For example, in a programmable network, the networks can be reprogrammed by injecting them with code (Section 11.7.8.3). Although service adaptation to networks can be done end-to-end, the ability to configure the network at the application layer is limited. A more flexible way of configuring the network is to modify the transmission and routing of data in the intermediate network nodes between the sender and the receiver. An example of applications controlling a network locally is configuring filters on specific types of traffic to block disruptive traffic or change QoS priorities. Due to the lack of support for application-specific routing in standard network elements at the network layer, this is supported at intermediate nodes in the application layer. Examples of this type of service-oriented network models are programmable networks, content-based networks, and overlay networks (Section 11.7.8). Raz et al. (2006) describe a system architecture for contextual network services based on the use of a programmable model. At the heart of the system is the service layer, which supports contextual service creation and policy-based service management. The service layer uses an underlying active application network platform to control and access an IP-based network. This system has been applied to multiple scenarios such as: B. A write-back scenario (Section 3.3.3.4) that waits until a high-bandwidth connection is available before transferring a large file; priority use of a network in a medical emergency and a virtual conference where the system evaluates the QoS of each participant's network connections before advising the participants on how best to configure their network connections for participation. A central design problem for the perception of the network is how network contexts are accessed and how they are distributed. Ocampo et al. (2005) define the concept of context flow, in which a flow defined as a network data flow following a defined network protocol is described using a (network) flow context. The flow context label information may be sent together with the flow itself and received by any intermediate network element that may act on it. This has the advantage that the flow context can affect short-lived network flows. While context awareness can improve information filtering, it can also do the opposite. For example, computer hardware and software increasingly know their version relative to the latest version released. It is becoming a daily routine for hardware manufacturers, operating system vendors, and all major application providers to distract users from tasks by automatically flagging updates. When updates are automatic, user control is reduced and systems can automatically reboot without the need to gracefully close applications. Also, these updates can rarely be avoided, they are often just delayed. Autonomy and knowledge of the status of many individual system components and the need to maintain them can be very distracting for users and active applications. A second big problem is that the context may be incorrectly determined and any resulting setting may be incorrect, e.g. B. Early SatNav systems occasionally led vehicles through blind alleys. Finally, user expectations must be realistic.

242

Context Aware Systems

The acquisition of non-deterministic environmental contexts, p. B. weather, or non-deterministic user contexts, such as a user's leisure activities, will cause automatic perception of the context to be inherently semi-deterministic or non-deterministic.

EXERCISES 1. Compare a general distribution system, an integrated control system, a sensor-based system, a general context sensitive system, a location sensitive system, and a custom system. 2. Outline the design of a simple state-based model for context awareness based on the current context state driven by pre-planned state transitions to bring it to the target context state. 3. Discuss issues of privacy and positioning ethics. Is it acceptable for a provider to be able to determine the location and context of a requester or customer when an incident occurs when they want to and vice versa? 4. Discuss design issues in system design to be aware of multiple contexts for an application of your choice. In particular, consider how you handle your design: conflicting and overlapping contextual information; whether multiple contexts fit in one stage or in an order where order matters; how to deal with uncertainty and how to deal with different semantics for contexts. 5. Discuss whether it is useful for systems to be aware of their external environment without being aware of themselves. 6. Compare and contrast different positioning methods in terms of accuracy, indoor and outdoor use, and local versus global positioning. 7. Discuss the motivation for ICT environmental awareness for mobile users. Outline designs to support context-aware adaptation of content to the properties of access devices and their local network connection.

References Baldauf, M., Dustdar, S. & Rosenberg, F. (2007) A survey of context-aware systems. International Journal of Ad-hoc and Ubiquitous Computing, 2(4): 263-277. Brown, PJ, Burleson, W., Lamming, M., et al. (2000) Context awareness: some compelling applications. In CH12000 workshop on what, who, where, when, why and how of context perception. Chen G and Kotz D (2000) A survey of context-aware mobile computing research. Technical Report TR2000-381. Available at http://citeseer.ist.psu.edu/chen00survey.html. Accessed November 2006. Clarke, K.C. (2001) An Introduction to Geographic Information Systems, 4th Edition Upper Saddle River, NJ: Prentice Hall. Couderc, P. and Kermarrec, A.-M. (1999) Improving service levels for mobile users through context awareness. In Proceedings of the 18th IEEE Symposium on Trusted Distributed Systems, pp. 24-33. Davies N, Cheverst K, Mitchell K et al. (1999) Airborne Caches: Information Diffusion in the Guidance System. In Proceedings of the Second IEEE Workshop on Mobile Computing Systems and Applications (WMCSA 99), pp. 11-19. Dey, A. K. (2000) Providing architectural support for building context-aware applications. Doctor Thesis. Department of Computer Science, Georgia Institute of Technology, Atlanta, November 2000. Available online at http://www.cs.cmu.edu/anind/context.html Accessed June 2007.

references

243

Dey, A.K. and above, G.D. (2000) Towards a better understanding of context and contextual awareness. In Proceedings of the Workshop on the What, Who, Where, When and How of Context Awareness. Dey, A. K. & Abowd, GD (2001) A conceptual framework and set of tools to support rapid prototyping of context-aware applications. Journal of Human-Computer Interactions (HCI), 16(2-4): 97-166. Djuknic, G.M. and Richton, R.E. (2001) Geolocation and assisted GPS. Computer Science, 34(2):123-125. Freeman, E. & Gelernter, D. (2007) Beyond Life Streams: The Inevitable Disappearance of the Desk Metaphor. In V Kaptelinin and M Czerwinski (eds) Beyond the desktop metaphor: Designing integrated digital work environments. Cambridge, MA: MIT Press, pp. 19-48. Gardner, H. ([1983] 2003) Moods: The theory of multiple intelligences. New York: Essential Books. Garnet, J. (2007) Geotools: The Open Source Java GIS Toolkit User's Guide. Retrieved June 2007 from the Geotools website: http://www.geotools.org/. Gelernter, D. (1992) Mirror worlds or everyday software puts the universe in a shoebox. . . How it will happen and what it will mean. Oxford: Oxford University Press. Gustavsen, R. M. (2002) Condor: An Application Framework for Mobility-Based Context-Aware Applications. In Proceedings of the workshop on concepts and models for ubiquitous computing, Gothenburg, Sweden. Gutta, S., Kurapati, K. & Schaffer, D. (2004) From Stereotypes to Personal Profiles via Viewer Feedback. In W Verhaegh, E Aarts, and J Korst (eds.) Algorithms in Ambient Intelligence. Dordrecht: Kluwer Scientific Publishing House. Hawkins, Southwest (1988) The Arrow of Time. In a brief history of time. New York: Transworld Publishers Ltd, pp. 143-154. Hazas, M., Scott, J., & Krumm, J. (2004) Location-aware computing comes of age. IEEE Computer 37(2): 9597. Henricksen K, Indulska J, and Rakotonirainy A (2002) Contextual information modeling in generalized computing systems. In F Mattern and M Naghshineh (eds) Proceedings of Pervasive 2002, Springer-Verlag, LNCS 2414: pp 67–180. Herring, J.R. (Eds.) (2006) OpenGIS Implementation Specification for Geographic Information - Simple Function Access - Part 2: SQL Option. Document No. OGC 06-104r3, version: 1.2.0. Available at http://www.opengeospatial.org/standards/sfs. Hightower, J. and Borriello, G. (2001) Location systems for ubiquitous computing. Computer Science, 34(8):57-66. Extended document: A Survey and Taxonomy of Location Systems for Ubiquitous Computing, available online at http://citeseer.ist.psu.edu/hightower01survey.html, accessed June 2007. Hofer, T., Schwinger, W., Pichler, Metal. (2003) Context awareness in mobile devices: the hydrogen approach. In Proceedings of the 36th Annual Hawaii International Conference on Systems Sciences, pp. 292-302. Kasanoff, N. (2001) Make it personal. New York: Persus Publishing House. Loke, S. (2006) Context-aware generalized systems. Auber Publications. Long, S., Kooper, R., Above G.D., et al. (1996) Rapid Prototyping of Context-Aware Mobile Applications: The Cyberguide Case Study. In Proceedings of the 2nd International Conference on Mobile Computing and Networks (Mobi-Com 96), pp. 97-107. Maes, P. (1994) Agents that reduce work and information overload. ACM Notices, 37(7): 30-40. Marcussen, C. H. (2001) Mobile data and m-commerce in Europe: revenue perspective of a mobile network operator, 1999–2003. August 2001. Available online at http://www.crt.dk/UK/Staff/chm/P_CHM.htm, accessed May 2007. Meng, D. and Poslad, S. (2008) A Reflective Context- aware System for Spatial Routing Applications. Paper presented at the 6th International Workshop on Middleware for Ad-Hoc and Pervasive Computing, Leuven, Belgium, December 2008. Mills, D.L. (2003) A Brief History of NTP Time: Reminiscences of an Internet Timekeeper. ACM SIGCOMM Computer Communication Review, 33(2):9-21. Morse, D.R., Dey AK & Armstrong, S. (2000) The what, who, where, when and how of context awareness. Workshop. Abstract. In Proceedings of the 2000 Conference on Human Factors in Computer Systems (CHI 2000), The Hague, The Netherlands, 1-6. April, p. 371. Ocampo R, Cheng L, Lai Z, and Galis A (2005) ContextWare Support for Network and Service Composition and Self-adaptation. MATA 2005: 84-95. Pashtan, A. (2005) User location and mobility management. In mobile web services. Cambridge: Cambridge University Press, pp. 54–80.

244

Context Aware Systems

Poslad, S., Laamanen, H., Malaka, R., et al. (2001) CRUMPET: Creation of friendly and personalized mobile services for tourism. In Proceedings of 3G2001 Mobile Communication Technologies, London, 2001, pp. 28-32. Prekop, P. and Burnett, M. (2003) Activities, context, and ubiquitous computing. Computer Communications, 26 (11): 1168-1176. Ratnasamy, S., Francis P., Handley, M., et al. (2001) A scalable content addressable network. In Proceedings of the Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, San Diego, California, pp. 161-172. Ratcliffe, J. H. (2001) On the accuracy of geocoded TIGER-type address data in relation to cadastral and census units. International Journal of Geographic Information Science, 15(5): 473–485. Raz, D., Juhola, A., Serrat-Fernandez, J., et al. (2006) Fast and efficient contextual services. New York: John Wiley & Sons, Ltd. Ryan N., Pascoe J. & Morse D. (1998) Augmented Reality Field Research: The Context-Aware Archaeological Assistant. In V Gaffney, M van Leusen and S Exxon (eds) Computer Applications in Archaeology. British Archaeological Reports, Oxford: Tempus Reparatum. Saha, S., Chaudhuri, K. Sanghi, D., et al. (2003) Determining the location of a mobile device using IEEE 802.11b access point signals. IEEE Wireless Communications and Networks, 3: 1987-1992. Samet, H. (1990) The design and analysis of spatial data structures, Reading, MA, Addison-Wesley. Schilit B., Adams, N. and Want, R. (1995) Context-aware computer applications. In Proceeding of 1st International Workshop on Mobile Computing Systems and Applications, pp. 85-90. Schilit B and Theimer M (1994) Dissemination of information from active maps to mobile hosts. IEEE Networks, 8(5):22-32. Strang, T. and Linnhoff-Popien, C. (2004) A context modeling survey. In the 1st International Workshop on Modelling, Reasoning and Management of Advanced Contexts. UbiCom 2004, pp. 34–41. Van Greunen, J. and Rabaey, J. (2005) Location and time synchronization services in ambient intelligence networks. In W. Weber, J.M. Rabaey and E. Aarts (eds.) Ambient Intelligence. Berlin: Springer Verlag, pp. 173–197. W3C CC/PP (2006) Composite Capabilities/Preference Profile (CC/PP) 2.0 home page. Available at http://www.w3.org/Mobile/CCPP. Accessed July 2007. W3C MMA (2002) Multimodal Interaction Activity, home page. Available at http://www.w3.org/2002/mmi/. Accessed July 2007. Want, R. (2006) Introduction to RFID technology. IEEE Pervasive Computing, 5(1):25-33. Want, R., Hopper, A., Falcao, V., et al. (1992) The Active Badge Location System. ACM Transactions on Information Systems, 10(1): 91-102. Watson, R. T. (2006) Spatial and temporal data management. In R. T. Watson, Data Management: Databases and Organizations, 5th Edition, Chichester: John Wiley & Sons, Ltd, pp. 415-430. Yang J, Yang W, Denecke M et al. (1999) Smart Sight: a tourist assistance system. In Proceedings of the 3rd International Symposium on Portable Computers, pp. 73-78.

8 Intelligent Systems (SI) With Patricia Charlton

8.1 Introduction Intelligent systems (IS) are systems that use artificial intelligence (AI), also called artificial intelligence or computational intelligence, and include agent-based (intelligent) systems, software agents, and robots. Many "smart" and highly flexible algorithms require iterations of refinement, sometimes involving manual input, to "mature" into algorithms that can be reused. There is also intelligence in the development process to build such an algorithm. The output of an intelligence system can be very complex and flexible, but it is often seen as just computing and not computational intelligence. For example, optical character recognition (OCR) used in scanners was previously considered part of computer vision AI, but is now only considered part of document scanning. This is what led Rodney Brookes of the MIT AI Lab to coin the term "AI effect" for algorithms that have matured and are no longer considered AI. When modeling systems that exhibit intelligent behavior, researchers often rely on studying how humans solve problems by using analogies consistent with the reasoning and knowledge processing that has taken place. This process allows us to think about how problem solving occurs, using the brain as a concept to test theories. It is clear that machines do not solve problems in the same way as humans. In fact, machines can be designed to be very good at solving problems that humans are not, and vice versa. There is also debate as to whether biological components, e.g. B. the approximately twenty billion neurons that make up the brain are individually intelligent or intelligent in groups and if other animals such as insects, e.g. B. ants or bees, whether or not they display intelligent behaviors or whether even simpler unicellular and multicellular organisms can be used as building blocks for intelligent systems. Surprisingly simple behavior can give rise to much more complex collective behavior that can be called intelligent.1 Such systems will not have a rich and explicit notion of knowledge. Systems built on this type of

Stefan Poslad

Intelligent Systems

246

The behaviors are called artificial life and are often the subject of many artificial science textbooks, e.g. B. Coppin (2004, pp. 363-383).

8.1.1 Chapter Overview This chapter is the first of three chapters that make up the Intelligent Interaction section of the book. Intelligent interaction is divided into three main chapters: individual intelligent systems of different types; Intelligent systems composed of intelligent entities with multiple interactions: Chapter 9); Autonomous interaction and artificial life (chapter 10) This chapter (section 8) focuses on the different types of individual intelligence proposed by Russell and Norvig (2003, pp. 44–54). This chapter continues with a discussion of basic concepts. The next major section (Section 8.3) focuses on architectural designs. Three main IS models and representations are then considered: semantic IS KB models (Section 8.4), classical logical IS models (Section 8.5), and Soft Computing models (Section 8.6). Some of the most important operations of the generic IS model are then discussed.

8.2 Basic Concepts 8.2.1 Types of Intelligent Systems Due to the variety of different types of AI and different properties of AI, no universal definition of AI is given here. Instead, some dimensions for classifying certain types of AI are discussed, and then some typical design models for intelligent systems. There are several dimensions into which AI can be classified (Box 8.1). Each of these is considered in more detail below.

Table 8.1 Dimensions along which intelligent systems can be classified Dimensions for classifying types of intelligent systems Strong or weak intelligence Physical (embedded) hardware, e.g. B. robots or virtual software, z. B. Software agents Basic properties such as autonomous, social, reactive, proactive, etc. Thought (cognitive) or action (behavior) Human or rational Complex organisms (explicit, knowledge-based, high-level choice of action) or simple cellular organisms (implicit, low-level (choice of action) Architecture type design: reactive, model-based, goal-based, utility-based, etc. Learning or non-learning Certainty or uncertainty The environments in which intelligent systems operate: observable, deterministic, sequential, etc. Individual intelligent entities or as entities multiple, collective and intelligent

Honey bees can use a “dancing language” to give directions to a field of flowers up to a kilometer away (Riley et al., 2005). This was first suggested by Karl von Frisch, who later received the Nobel Prize in 1973. Ants can regulate nest temperatures within 1°C, cooperate to transport large objects, and find the shortest routes from their nest. nest to a food source.

basic concept

247

There is a philosophical difference between the notion of weak AI, where machines can act or simulate intelligence, and strong AI, where machines actually think. An alternative view of strong and weak intelligence is to define a set of sub-traits for intelligence and group them into strong or weak. For example, Wooldridge and Jennings (1995) define two types of intelligent systems, called weak agents and strong agents. A weak agent is defined as supporting system properties such as autonomous,2 social, reactive, and proactive. A strong agent is defined as mobile, truthful or honest, benevolent (an agent will always try to do what is asked), and rational. A distinction can be made between an intelligent system that is physically embedded, e.g. a robot that acts locally and an intelligent agent that exists in a virtual computing environment, e.g. B. a software agent that can act globally and is free (remote) within the entire space of the virtual computer, d. h Internet, acting and wandering. Various other researchers have classified the properties of IS as follows. Nwana (1996) identifies three central properties of agents: autonomy, learning, and cooperation, and then seven types of agents such as collaborative, interface, mobile, information/internet, reactive, hybrid, and intelligent in terms of their support for his three properties. central. Franklin and Graesser (1996) describe the central characteristics of agents as reactive, autonomous, proactive, continuous over time, communicative, adaptable, mobile, flexible (using unwritten behavior), and humane. Other commonly used classifications are mobile vs. static intelligent agents, reactive vs. deliberative agents, and individual vs. multi-agent systems. Each of these intelligent entity classifications overlap to some extent, but there are important differences. Partly to address the difficulty with the strong versus weak type of AI classification and the inherent difficulty of distinguishing between simulated thinking and real thinking, Russell and Norvig (2003) distinguish types of intelligence3 along two dimensions, first in relationship with the action of systems (behavior) versus systems thinking. Secondly, types of intelligence are distinguished depending on whether human intelligence is called systems that act or think like humans (physical, mental, emotional, moralistic, etc.) or systems that think and do what is correct or ideal, which They are called rational intelligences. . Rationality can be more accurately defined as achieving a defined measure of performance given what you know about your environment and the effects of your own actions, as well as the current and past states of the system and environment. There are different designs to support rational behaviour, for example based on reactive behavior (Section 8.3), internal systems models (Section 8.1.1), goal-based functioning and utility (Section 8.6) and learning as proposed by Russell and Norvig (2003, pp. 32-54). Action often involves thinking about doing the right thing.

8.2.2 Environment types for intelligent systems It is a challenge for all systems to operate in open system environments. Russell and Norvig (2003) categorized open systems environments4 according to several dimensions (see Table 8.2). The simplest types of system environments are fully observable, episodic, and static. From the system point of view, static means that the environment does not change during the selection and execution of actions to influence the system. Environments change state naturally

The link between autonomy and intelligence is explored in more detail in Section 10.2.1.1. This ranking of human vs. rational AI also overlaps with that of strong vs. weak AI. Strong AI is AI that matches or exceeds human intelligence. Weak AI refers to the use of software to mimic certain subtypes of human intelligence. 4 While the classification is specific to intelligence system environments, it can also be applied more generally. The term "world" is often used in place of environment. 3

Intelligent Systems

248

Table 8.2 Environment models for UbiCom systems based on Russell and Norvig's (2003) classification of environments for intelligent systems Environment type

environment description

anonymous environment

Fully visible, accessible

UbiCom sensors provide access to the full state of the environment at all times

Partially visible, accessible

deterministic

The next state of the environment is completely determined by the current state and the action taken by the UbiCom system. If the environment is deterministic except for the actions of other UbiCom systems, then the environment is strategic

stochastic, not deterministic

episodic

The experiences of the UbiCom System are divided into atomic "episodes". Each episode consists of the system detecting a single action and then executing it. The choice of plot in each episode depends only on the episode itself.

sequentially

Static

The environment remains unchanged while a UbiCom system selects and executes its actions, ie. h adapts to its environment

Dynamic

Discreet

The environment is characterized by a limited number of clearly defined states and actions.

Continuously

Passive

The UbiCom system environment can be dynamic, stochastic, etc., but the environment itself is not active in the sense of modeling the system that operates within it.

Active

in terms of time, perhaps due to the functioning of the active systems that act on it. More complex designs of intelligent systems are required to think and act in environments that are (1) uncertain and nondeterministic; (2) viewed or captured by the system in part rather than in full; (3) they are sequential and use the history of events to predict future actions instead of relying solely on current actions; (4) they are dynamic rather than static; and (5) consist of other smart components instead of being surrounded by stupid passive components. Most open system environments tend to be stochastic, since no single system controls the environment. The actions of many other active systems can cause actions and changes in the environment that affect each individual system. Of course, the environment itself can be stochastic, e.g. B. the weather "system" in the physical environment.

8.2.3 Use of intelligence in ubiquitous computing The use of the term intelligence varies in different computing areas. Ambient intelligence (AmI) is used to refer to intelligent environments, but it is more of an abstract model than some kind of intelligent system specification (Lindwer et al., 2003). Updates suggested by Ramos et al. (2008) to specify a more explicit AI model for AmI adds a layer of intelligence, but there are no specific architectures. Human (intelligent) computer interfaces can be designed to accommodate human users (Chapter 5). AI can be used to power controllers and programmable robots (Chapter 6). AI can be the basic design for contextual systems: systems that perceive and act on the environment and adapt to the environment (Chapter 7). The AI is connected to programmable networks, called smart grids, which put intelligence, perhaps in the form of logic, into network switching components rather than the end-use computers. Agent-like themes can be used for network administrator

IS architectures

249

processes, e.g. B. SNMP Agents. Here it is proposed to use specific types of IS architecture models from Russell and Norvig (2003) as a basis for specific types of IS design for UbiCom. To explain the utility of UbiCom's IS-based designs, some applications and scenarios described above in Section 1.1.2 are re-examined as to how they might benefit from the use of AI. For the Personal Memories scenario (Section 1.1.2.1), an IS represents the AV system. Its design uses artificial vision to detect eyes and mouth, to determine if someone is smiling or not, to detect if the recorded faces match already classified faces. Metadata (data describing other data) to annotate the image, e.g. B. using the time, place, identity of registered persons, etc., and all visual characteristics extracted from the image can be represented as explicit knowledge. A system to support this scenario could be based on a reactive IS design. For the adaptive bus scheduling scenario (Section 1.1.2.2), an IS representing the bus service can use utility functions to weight the importance of several independent factors that must combine to support an objective, e.g. B. Pick up more passengers to generate more revenue instead of picking up fewer passengers to minimize the level of delay of a late bus. The prediction of the arrival of a bus at the scheduled bus stops is based on a model, such as the environment of a bus, such as the city. B. road conditions can dynamically affect the speed of the bus along the route. A system to support this scenario could be based on a model-based agent and service-based IS design. For the feed management scenario (Section 1.1.2.3), the IS represents the feed management system. Requires the use of knowledge transfer to deal with the different conceptualizations used on food labels versus a recipe instruction book, e.g. For example, the recipe may refer to zucchini weighing in pounds, while the product label may refer to zucchini weighing kilograms; these terms are similar. A robot requires adaptive sensing and control techniques to plan and execute its food packing and unpacking operations in grocery stores. Events must be monitored and evaluated based on predetermined rules. A system to support this scenario could be based on a combined reactive and goal-based IS design. For the provisioning regulation scenario (Section 1.1.2.4), the IS system recognizes the user context and adjusts system usage based on the user context, a model of the user's expected usage, and user management policies. A system to support this scenario could be based on goal-oriented and reactive IS design.

8.3 IS Architectures An IS system is based on a traditional distributed system architecture with three extensions. First, it supports richer interaction models between ISs and their environments, such as B. Reactive, Proactive, and Benefit-based interaction models. Second, it uses processing models that support a model of cognition and advanced systems behaviors such as searching, reasoning, planning, and learning. Third, it uses a richer communication model with other intelligent systems and with intelligent environments (Chapter 9).

8.3.1 What a model knows versus how it is used IS models can be classified in several ways: by the type of architecture of the model, i. h according to what the model represents or what the model knows, the epistemological level or the structural level; how the model is used to solve a problem, the heuristic level, p. B. as a logical language that supports reasoning and heuristics to confirm hypotheses and derive new knowledge; what environment a system is in and can be operated in (Section 8.2.2).

Intelligent Systems

250

McCarthy and Hayes (1987) separated the representations of the model first at the epistemological level and then at the heuristic level. A representation is said to be epistemologically adequate for a person or a machine if it can be used in practice to express the facts one really has about that aspect of the world. A representation is called heuristically adequate if the thought processes actually used to solve a problem can be expressed in this representation. Typically, there is a trade-off for representation between representational power at the epistemological level and inference efficiency at the heuristic level. When discussing IS design, many AI practitioners often assume that a specific logic-based representation is used for the IS environment model. This focuses on a heuristic rather than an epistemological explanation. The logical model is often referred to as the deliberative or reasoning-based IS type. Semantic knowledge-based (KB) systems models focus more on the epistemological level to make them reusable in many different applications. In practice, in order to use a general epistemological level knowledge base, additional knowledge relationships and processes must be modeled to relate the general knowledge base model to the specific application domain. If the KB is an ontology, it refers to the application-based ontology assertion specification. A learning-based IS can be used to acquire any type of model used in any type of IS, the structural representation of knowledge depends on the type of IS. For example, a learning-based SI can be used to capture the environmental context. This often uses a model representation that can handle uncertainty and heuristics for knowledge acquisition, so it focuses more on KR at the heuristic level. Often it may not be clear what the model represents, e.g. for example, the model could represent the design of the internal system, the design of the external environment, or both. In this text, two distinct dimensions of model or knowledge have been separated in terms of what the model represents and how the model is represented in computational form.

8.3.1.1 Types of architecture models Russell and Norvig (2003) have proposed several main types of IS architecture models, depending on how system actions are controlled and what type of environments they run in (see Table 8.3). In reactive systems, the actions of the system depend only on the observed current.

Table 8.3 Intelligent system designs in relation to the types of environments for which they are suitable in the IS model type

What the actions of an IS depend on (what is modeled in the system)

Types of environments for which IS design is appropriate

reflex based

Context of the current environment

fully observable, stochastic, episodic, static, physical

on situated action based on models

Current and past environmental context

Partially observable, deterministic, sequential, dynamic

Goal oriented, proactive

YES action plans to achieve a goal

Partially observable, deterministic, sequential, human dynamic

utility based

Weighting of different IS objectives and plans

Partially observable, semideterministic, sequential, dynamic

Learn

Performance

Partially observable, non-deterministic, sequential, dynamic

IS multiple

Partially observable, non-deterministic, sequential, dynamic

IS architectures

251

environmental context. Environment model-based systems5 are a design for IS that allow systems to consider past contexts and models of how the environment works in order to anticipate how to most appropriately act in changing environments. The actions of a goal-based IS depend on a system's model of how its processes and actions lead to a result, which allows a goal-based IS to be more proactive. In utility-based IS systems, actions depend on how action options and goals are weighted. In learning-based IS designs, system actions can be adapted to improve internal performance metrics over time. With multiple IS designs, the actions of each individual system may depend on active interaction with other autonomous IS systems, as well as any internal system models. In practice, individual designs are often combined into hybrid systems, e.g. B. Designs of reactive environmental model systems and goal-driven reactive environmental model systems. Designs of multiple interacting systems may use multiple hybrid individual IS designs.

8.3.1.2 One-sided versus two-sided system environment models In general, when designing pervasive system applications, a one-sided environment model is used, e.g. For example, a context-aware application can model its environment, but not the other way around. The environment is not active. However, as we move towards more intelligent environments, they can be designed to contain a model of the application systems that reside in or traverse them (Figure 8.1). A system that models an active environment, which in turn has a model of the systems that use it, can lead to much more complex and cyclical dependencies between multiple systems and environment models. System designers may need to consider the level of intelligence of the environments, e.g. people, other UbiCom systems, etc., and how well the environment can model and understand the systems they inhabit.

Action Selection Passive Environment Intelligent Environment

Incoming event sensors

Intelligent System

Output-Events-Effects

action selection

Incoming event sensors

Intelligent System

Output-Events-Effects

action selection

Figure 8.1 One-sided active system model (left) compared to two-sided active system and active environment models

5 Russell and Norvig (2003) refer to one type of (intelligent) agent-based systems as model-based systems. The term based on models is considered here too general and is relativized with the term environment, since model refers more precisely to the model that a system has on the behavior of its environment.

252

Intelligent Systems

8.3.1.3 Model representations There are several main concrete representations for IS models and a variety of model operations that can be performed that depend on these representations. First, IS models can be represented as process-driven system models in which prescribed courses of action are specified that represent a process at design time. It can be difficult to correctly design and maintain such processing algorithms for complex systems. They may also require the use of extensive computing resources. Computational process models were discussed in detail in Chapter 3. Second, IS models can be represented as data to support a variety of syntactic and semantic conceptualizations, i. h knowledge-based models (section 8.4). Third, IS models can be represented using different types of classical logic (Section 8.4.2.5). Fourth, IS models can be represented using different types of computation (Section 8.6). These model-based representations for IS are discussed in the subsections that follow.

8.3.1.4 How system models are acquired and adjusted Nothing has been said so far about how an IS obtains a model of its own operation and internal system environment. There are two main ways to design how the model is captured. First, system models can be created by a human designer and incorporated into the system at design time. The limitation here is that the designer may not be or have much experience with the environment in which the system will be used, or usage may vary between user environments. As such, a design may have a fixed mode of operation, may be incomplete, or may not have the ability to be configured for operational use. To meet such needs, systems can be designed to be regularly maintained and updated via remote access links from a service repository (Section 12.3.2). Second, systems can be designed to acquire their own models of self-operation and models of their environment: systems that learn. These systems are also known as machine learning systems. There are several machine learning-specific designs (Section 8.3.6). Hybrid model acquisition by systems processes can combine manual design and maintenance with machine learning. Often the adaptability property of an IS is taken as synonymous with the learning property, but adaptability or flexibility is a more general concept, learning is a specific type of design to support adaptability. A goal-based design that supports multiple plans can also allow the system to adapt in the sense that it can switch to a different plan if the environment causes one plan to fail. Another type of matching is based on a reflection type system design (Section 10.3).

8.3.2 Reactive IS Models In a reactive system, intelligent behavior arises from the interaction of the system with its environment rather than from complex internal knowledge representations or inferences about events. The choice of action6 is determined exclusively by the current state of an environment. Rules or conditions are used to filter input events and determine which system actions are triggered in response to specific events7. A reactive type IS is deeply embedded in its environment and responds very well

A reactive system is also called a reflex system, perception-based system, or auxiliary reactive atomicity. 7 Stock selection is the heart of the intelligent system. An action selector is also known as a controller.

IS architectures

253

Environment exit events

Input events Sensor Current environmental context

Environment input events

Exit Events Level 3 Event Loop

Effector Layer 2 Event Loop Rule Action Selection

Layer 1 event loop

Smart system Simple smart system with a single event loop

Complex intelligent system with multiple event loops

Figure 8.2 Reactive Type Intelligent System

to changes in the environment. Reactive systems are typically designed as event-based systems (Section 3.3.3.6) that go through the basic operations of capturing events, filtering events, and then triggering system actions (Figure 8.2). Preset actions can be directly triggered by unconditional sensor inputs, e.g. B. In the adaptive transportation planning scenario, when the vehicle arrives at a specific passenger pick-up point, it will stop automatically. Alternatively, events can be filtered by conditions to trigger actions, e.g. For example, a vehicle will stop at a particular stop only if the passengers in the vehicle have requested to exit the vehicle, or if one or more passengers are waiting at the stop and request that the vehicle stop. A reactive system is a natural design for simple sensor control systems (Chapter 6) and for simple context-aware systems (Chapter 7). In a simple reactive system, where there is only one event loop, only one event can trigger an action at a time. More complex designs must consider how to respond to multiple simultaneous events and how to handle multiple actions that may conflict when triggered. There are three possible designs for handling multiple, possibly heterogeneous, concurrent events: (1) drop events that cannot be handled in time; (2) support of persistence of events; and (3) support concurrency event handling. The first option is the least desirable because it drops events non-deterministically. However, it may be necessary to delete events if unforeseen events occur that cannot be processed. Second, drafts can support event persistence, such as the use of event buffers or heaps when action selection decisions can be deferred in response to some events (Section 3.3.3.6). Third, designs can support concurrency by incorporating multiple types of event loops and overlapping event loops and prioritizing different event loops to trigger actions (Figure 8.2). For example, in the adaptive bus scheduling scenario, the event "passengers at the bus stop" triggers the "pick up passengers" action as priority over the "drop off passengers at the bus stop" action because the action is "the bus he comes late". The purely reactive type of intelligent system works best when in environments that are fully observable, static, and episodic. In practice, many systems are designed not to be purely reactive, but to combine reactive behavior with other behaviors, such as model-based behavior, goal-based behavior, and utility-based behavior. These are types of hybrid reactive systems. Hybrid IS designs are discussed in more detail in Section 8.3.7.

Intelligent Systems

254

8.3.3 Information system based on environment models An effective way to deal with partial observability of the environment is for a system to use an internal model of the environment, perhaps by tracking past events and analyzing them for patterns and knowledge of those events. effects of the internal uses of the system. actions (Russell & Norvig, 2003). This environment model is also known as the world model. It is a kind of knowledge-based system because the system has knowledge about the world and its actions. The intelligent system8 environment model type can model sequential or historical behavior in the environment. It supports predictive atomicity, making decisions based on current events and a model of past external events that can be used to predict future external events (Figure 8.3). The IS environment model uses a sequential environment model. These types of IS actions depend on the current environmental state, past environmental states, and knowledge of the impact of system actions. This type of systems design is similar to a situational action type of systems design: actions may be unplanned and highly context dependent (Suchman, 1987, pp. 49–67). It is also possible to anticipate multiple future environmental states that may never be realized, leading to a theory of multiple possible future environments or world states. For example, in the adaptive transportation planning scenario, if someone appears to be running to the pickup point but has not yet arrived, an environment model-based IS expects that when the vehicle slows down or waits for a while, it will arrive. an additional passenger expected. at the pick-up point, to be promoted. In the purely (conditionally) reactive system, the vehicle will not stop if there are no passengers at the pickup point (as long as no passenger in the vehicle wishes to disembark).

IS design environment exit events based on the hybrid environment model

Incoming event sensors

effects action effect

Send to Kontext

Stock Selection Rules

about model

Context of the current environment

event loop

action selection

Intelligent system Figure 8.3 SI based on environment models according to Russell and Norvig (2003)

Russell and Norvig (2003) refer to this design of an IS as a model-based reflex agent, refer to an IS as an agent, and emphasize that the reflex agent design is extended by adding an environment model to the system.

IS architectures

255

Systems that create such an environment model allow their services and applications to optimize and adjust their behavior to account for behaviors in the environment that are inaccessible but default (as defined in the environment model). A limitation of the pure environment model type of IS is that it does not contain an internal behavior model, e.g. B. Action processes through the system.

8.3.4 Goal-Based SI A goal-based SI, also known as planning-based SI, defines an internal plan or sequence of actions to achieve a future goal of the system (Figure 8.4). Unlike environment model-based IS, the choice of action for goal-based IS depends on what is the system's next best action to move the system toward a future target state. Compared to the IS based on the environment model, a goal-based IS tends to separate the control of actions from the environmental situation or the context of the action, in contrast to a system based on a reactive environment model when events trigger system actions like external events. In goal-based systems, internal events, e.g. a delayed scheduled system task, trigger system actions. The system can also trigger actions when external events are detected: active detection of external events. The main benefit of this type of design for users is that users can delegate tasks to this type of IS at a much higher level of abstraction, focusing on what needs to be accomplished rather than the lower-level details of how to do it. will achieve this. . In practice, the distinction between a target as a state and as a high-level action is similar, e.g. B. The target state of a system normally off versus the high level action to shut down a system normally. A key design issue is how a user

Basic objectives or SI based on benefits

Objective-based or benefit-based hybrid IS (which also includes behavior based on environmental models)

Surroundings

Incoming event sensors

Send to Kontext

Environment exit events

Intelligent System

effectors

objectives or benefits

Incoming event sensors

Send direction Context Context of the current environment

action selection

next env. context

output events

Intelligent system action effect.

effectors

objectives or benefits

on Selection of the model of action

Figure 8.4 Two types of goal-based or utility-based IS design: basic versus hybrid, from Russell and Norvig (2003)

256

Intelligent Systems

of an objective-based system knows what objectives a system can achieve. A goal-based IS can post what goals you can achieve and your action plan to achieve those goals. A hybrid goal-based IS combines the goal-based action choice of the basic goal-IS design with a world environment or model. Such an SI design selects actions not only based on the (external) environment, but also based on which chain of actions will allow the system to reach its target state (Figure 8.4). For example, in the adaptive transportation planning scenario, a goal for the service provider might be for the transportation vehicle to arrive on time at each specified pickup point. Therefore, a vehicle may choose to ignore a passenger who is on his way to a passenger pick-up point but is not yet there, because if the vehicle were to wait, it may not be able to meet its pick-up obligations. passengers in a timely manner to meet up later on the route. Finding action chains that allow an IS to reach its target state requires research and planning (Section 8.1). Some goal-based themes may support multiple plans. This allows the system to adapt in the sense that if the environment causes a plan to fail to achieve a goal, the system can easily switch to another plan. Goal-based design can also involve cooperation and goal sharing with other information systems (Chapter 9).

8.3.5 Benefit-Based IS A benefit refers to a quantifiable measure of the performance or value or utility of a particular goal out of a set of possible goals. It can also refer to a specific chain of actions from a set of possible chains of actions. A utility function is used to assign a state (goal) or a string of states to a value that represents its performance or value. A utility-based IS design makes sense when there are multiple conflicting goals, or when multiple goals are possible but only one of them is practical or achievable. For example, for the adaptive transportation planning service, two conflicting objectives9 for recovering from schedule disruptions are maximizing revenue by maximizing pickup load versus maintaining service quality by minimizing deviation from a schedule. For example, the larger the load collected, the later a vehicle that has already been delayed will arrive. A utility function here might weight revenue generation and on-time maintenance equally, 50-50%, or it might weight revenue generation more than on-time maintenance, 75-25%. A benefit-based SI design looks similar to a goal-based SI design. A goal-based IS sets a specific goal and chooses the action that leads to that goal (goal-based action selection), regardless of its utility. A utility-based IS compares the utility of different possible outcomes and selects the action outcome with the highest utility (utility-based action bias). Sometimes it may require some phase of experience or training to set and adjust utility function values, therefore goal-based action selection may be preferred over utility-based action selection in a new environment.

8.3.6 IS based on learning Learning refers to a system that improves its performance with experience in a task (Mitchell, 1997). A system is said to learn from experience E with respect to a class of actions A and a performance measure P if its performance on the set of actions A measured against P improves with experience E. An example of the adaptive management vehicle of Goods: A logistics vehicle

There is a modeling option to represent something as a single multi-valued target or to view the same thing as multiple single-valued targets.

IS architectures

257

Learning IS design involves using the problem generator to suggest new actions and the critic to provide feedback.

prior knowledge

Environment exit events

input events

sensors

critical

Experiences with intelligent systems

effectors

performance measurement feedback

Slowly

output, predict action

Knowledge Base (KB) Knowledge Hypothesis Learning Item

hypothesis

action selection

learning machine

Behaviour

improvements

Action Learning Effect Objectives New problem generator

Learning IS Design is used to generate hypotheses for expected results and actions. The design dispenses with the use of the problem generator and presents the reviewer as a KB.

observations, experiences

Performance Element (Action Selection)

Figure 8.5 Two different learning IS designs, the one on the left according to Russell and Norvig (2003) and the one on the right that focuses more on using a KB and learning to generate hypotheses or heuristic functions

picks up goods on route", E¼ "route run", P¼ "deviation of actual time from planned time". The improvement is that the dimension P is reduced to zero. A basic architecture for a system is shown in Figure 8.5. based on learning, also called machine learning, according to Russell and Norvig (2003).They use a theater metaphor to illustrate how a basic learning system works.A learning system includes four main components: (1) a performance element; ( 2) a comment or criticism generator, (3) a learning element, and (4) a problem generator. The performance element behaves like the (external) action selection element in the other IS based architecture models. in experience, E, the combined input from the sensors, the learning element, and the problem generator A reviewer or feedback generator provides feedback on how well the agent is performing on function of a fixed performance standard or measure, e.g. For example, in the transportation system scenario, the reviewer provides information about the deviation of the expected arrival time at predetermined points on the route from the actual time, or asks passengers about their travel experience, including the punctuality and capacity, through a questionnaire. The learning element receives feedback from the critic and modifies the performance element, e.g. B. If customer feedback indicates that the shuttle was too busy, the shuttle may operate higher capacity or more frequent vehicles. The problem generator provides the performance element with additional input hints for new actions to be taken, otherwise the behavior of the system never changes, e.g. B. Try different routes to pickup points to improve punctuality.

8.3.6.1 Machine learning design The design of the learning element depends on what (which model) is being learned, the type of feedback, and the model or knowledge representation. Learning may require model representations that can do this.

258

Intelligent Systems

Dealing with uncertainty, which is often at the heart of learning technology. In general, three main types of learning or feedback can be used: (1) supervised learning; (2) unsupervised learning; and (3) reinforcement learning. Supervised learning, SL, also known as example programming, involves learning a heuristic function or hypothesis that converts inputs to outputs given examples of inputs and their associated outputs. Then predictions of the output can be made based on the input. If the actual output differs from the expected output, the heuristic function is adjusted. The process is repeated until the SI makes accurate and repeatable predictions of the output. Specific types of SL include inductive learning (see Figure 8.5), backpropagation neural networks, and decision trees. Inductive learning is all about learning a good heuristic function that maps input to output. Decision trees are learning to classify outcomes or actions, such as classifying a user's activity based on observations of weather and active devices. Unsupervised learning (UL) involves learning to distinguish input patterns without defining outputs (classifications). An unsupervised learning system can learn to discriminate patterns, but not what the desired outcome or classification is. Examples of UL are probability analysis, data clustering techniques (Han and Kamber, 2006), and Kohonen mapping neural networks. An example is looking for patterns in the context data and then classifying these patterns and therefore the contexts. A simple data clustering algorithm involves finding the frequent sets of things (Things can represent the types of AV content that people watch frequently, for example, identifying groups of things that also support a minimum count for confidence11. Learning by reinforcement, also called reward-based learning, involves evaluating actions based on rewards for performing an action, eg Q-learning Learning techniques can be used to acquire any knowledge for any of the models in any of the architectures of the the system, considering the number of possibilities of the learned model and presenting the learned information Otherwise, learning systems that acquire all their knowledge from scratch may take too long to acquire enough Acquire knowledge to become useful SIs can be conceptualized in this way form It may contain the so-called conoc background knowledge or embedded knowledge acquired or transferred from other IS systems or people, rather than being learned from scratch. ISs can then add knowledge through their experience by adding so-called foreground knowledge (Figure 8.5). As the dimensionality of the data increases, the amount of data needed to make good predictions increases exponentially.

8.3.7 Hybrid IS Hybrid IS models aim to combine the advantages of each IS model. There are two basic designs: horizontal concurrent levels versus vertical (sequential) levels (Figure 8.6). Layers consist of single or multiple IS components with a clearly defined interface for input and output. An example of a design to handle multiple simultaneous events has already been given (Section 8.3.2). A horizontal homogeneous layer model is used so that multiple events can be processed in parallel.

10 Support is the percentage of all samples where, for example, two specific things happen, e.g. B. A person's viewing history shows that he has watched comedy and drama. As a function, it is Support(A,B) ¼ P(A[B), where P is the probability. 11 Confidence is the percentage of times a person did one thing at the same time they did another (related) thing, p. As a function, it is confidence (A,B) ¼ P(B|A).

IS architectures

259

Environment Input Output Events Events Level 3 Target Model

Environment input events

Exit Events Level 3 Objective Model

Layer 2 enclosure. model

Layer 1 reactive model

Hybrid Vertical Layer Model Design

Horizontal Layered Hybrid Model Design

Figure 8.6 Two different designs for a hybrid IS based on horizontal and vertical layering

This type of model only needs to be generalized to allow stratification of heterogeneous models. For example, a hybrid IS can handle reactive events at a lower reactive layer and handle events that require environment model use and reasoning at a higher layer. A design challenge with this type of model is that since each layer emits its own action independently, multiple output action events can occur for the same input event. This requires an intermediary to coordinate and control the output actions (Wooldridge, 2001, pp. 89-104). A utility function or cost heuristic design could be used for a mediator, allowing the action with the highest utility to be generated when multiple actions could be triggered. Heterogeneous models for a hybrid IS can also be superimposed vertically instead of horizontally. In the drafts proposed in the previous sections (Sections 8.1.1, 8.3.4), several clearly defined IS models are linked. For example, in environment model-based IS, environment events are first processed by the environment model before being processed by the reactive model (ECA) (Figure 8.7). In goal-based hybrid designs, control first flows through a component or layer of the environmental model before moving to a goal layer. This model chaining is a type of one-step vertical model design, where control flows through each layer to generate action on the last layer. Other types of vertical model design may use multiple passes or flows for control,

IS design environment entry events based on a hybrid environment model

output events

Entry events of the IS design environment based on the hybrid target model

output events

(Video) BCIS 5379: Chapter 6: Mobile Commerce and Ubiquitous Computing

Layer 2 enclosure. model

Layer 1 reactive model

Layer 1 objective model

Figure 8.7 Simplified Layered Views for Model-Based IS Design in a Hybrid Environment and for Goal-Based Hybrid IS Design

Intelligent Systems

260

Generation of information and action. Note that this type of hybrid chained model does not allow for concurrent event processing tasks and can create a processing bottleneck. If one component fails, the entire chain could fail without the design being able to prevent it. Wooldridge (2001, pp. 89–104) discusses some concrete examples of horizontally layered IS architectures, p. B. Innes Ferguson's TouringMachines and Vertically Layered IS Architectures, p. InterRaP by B. Jörg Müller.

8.3.8 Knowledge-based (KB) Knowledge-based (KB) IS models encompass a set of IS models in terms of representation, operations, and what the model is used for (the model type). Common types of KB system architectures include production systems or rule systems, blackboard systems, and semantic KBs such as ontology-based systems (Section 8.4). Currently, the terms knowledge-based, semantic, and ontology-based models are often used interchangeably due to the prominent use of ontology-based models and their support for rich semantic conceptualization. Noy and McGuinness (2001) summarize the following benefits of KB models: (1) sharing a common understanding of information structure between humans and machines; (2) to allow reuse of domain knowledge; (3) make domain assumptions explicit; (4) separate domain knowledge from operational knowledge; and (5) analyze domain knowledge.

8.3.8.1 Production or rule-based KB system In a production system, knowledge is represented as a set of rules or productions stored in the KB. A rule engine based on logical reasoning is called a rule reasoning engine. A rule engine determines how rules constructed for an IF fact (also called a condition part or antecedent part) and an THEN fact (also called a consequent part or action part) are used. A rule-based KB model can be combined with a reactive IS. When new (environmental) events are generated, they are presented as new facts (things that are true). The KB system takes the fact in the IF part of the rule and compares it with the actual facts contained in the memory part of the KB (Figure 8.8). If a match is confirmed, the action rule is triggered and its THEN statements are added to memory. New facts that are added to working memory can also cause other rules to be triggered. Searching the knowledge base may involve forward or backward searches (Sections 8.7.1, 8.1.1). Rules can be added to the KB manually or automatically per machine

Rule Engine Step 3

The selection point is captured (New event, Done)

knowledge base

IF the people at the pick-up point THEN stop the transport vehicle If the transport vehicle stops THEN they pick up the people Working memory

Paso 1

People at the collection point Transport vehicle stop People collection Step 5

Figure 8.8: A rule-based IS based on knowledge

Step 2 Step 4

IS architectures

261

Learn. Another complication is that new data and searches may conform to multiple rules that may conflict. Methods for resolving rule conflicts include: take the first match; use more specific matches instead of more general matches; Using a higher priority rule over a lower priority rule; Using the last match or using logical inference. Many rule-based engines have been developed, some of which allow rule-based systems to be integrated as part of larger distributed systems rather than part of a more specialized IS, e.g. B. JESS, Java Expert System Shell, developed by Friedman-Hügel (2003).

8.3.8.2 Blackboard KB System The basic principle of a Blackboard KB system is that it acts as a shared knowledge repository among various possibly distributed processes (Section 3.3.3.7). Knowledge sources can be independent and heterogeneous, and unlike a reactive IS design, the repository allows selected events to remain stored rather than being removed from the event queue once consumed. A Blackboard KB can participate in multiple event condition asynchronous action rules. A Blackboard KB can be combined with a Rules-Based KB to allow multiple Rules-Based KBs to interact with the whiteboard. One of the limitations of a rule-like knowledge base is that the same fact can be stored in many rules. This makes it difficult to maintain the facts and their consistency. In addition, the relationships between the main concepts can only be expressed by rules. The rules may not be very good at expressing rich relationships and conceptualizations of facts. If the main requirement is to maintain and manage descriptive semantic relationships, then it makes more sense to structure concepts not as facts in rules, but in rich conceptualizations based on framework-based KBs and ontology-based KBs (Section 8.4).

8.3.9 SI models applied to UbiCom systems In the introduction, in the section on common myths for UbiCom (section 1.5.2), it was said that UbiCom systems do not need to be based on AI. On the other hand, others like Thompson and Azvine (2004) argue that AI is essential to enable UbiCom. The reasoning why AI is useful but not essential to enable UbiCom is that it depends on the type of application in the first place. Second, it depends on the specific AI model used, and third, there are some alternative computational models to AI that do something similar, e.g. For example, a system may use a simple event-driven system to sense its environment and support context awareness, and thoughtful system design implemented using a procedural language may allow a system to adapt to its environment. Specific IS architectures can be used to create systems that support other UbiCom system features, such as context awareness, autonomy, and iHCI, as follows. A reactive-type IS design is a good design for a minimally context-aware system, in which the actions of a system depend only on the current surrounding context or on well-defined changes in context that satisfy a given condition. In a more advanced context-aware system design, actions can also be designed to depend on the environment model (Section 8.1.1), including previous context switches. In practice, context-aware system designs also need support for assembling multiple context events, associating sensor events with applications, and managing contexts, including their storage and retrieval (Chapter 7). A context-aware (environmentally) system may also need to be aware of its own internal system actions in order to accurately control and act on its environment (Section 10.3). Some changes to the environment context, e.g. B. to support iHCI and user context changes, but are less defined and clear. For example, in the personal reminders scenario, consider a system that customizes the recording of audio and video content, you may not be able to unequivocally determine what

Intelligent Systems

262

user context, but only to give an estimate of it. In this case, an environment model ES that either renders the model using soft computing techniques or uses a hard computation that adds variable values from the environment context into a best estimate, e.g. B. collaborative filter systems to be a suitable design. Simple control systems could be based on the reactive and supply type IS designs. IS designs focus on selecting actions from a set of possible actions, whereas embedded control systems often have a very limited set of actions. The control systems focus is on fine granularity of control and the use of specific feedback control mechanisms that are not explicitly presented in any of the given IS designs. Therefore, standard built-in environmental control systems tend to use specific designs (Section 6.6). More complex controls could use knowledge of uncertainty models. Complex spatial control systems, i. h Robots must contain spatial sensors and spatial determination and estimation algorithms to move in prescribed ways, along with collision detection (reactive IS behavior) and a spatial memory to recognize obstacles and remember how to do so in the future to avoid them (model behavior environmental). Autonomous systems are similar to IS in that they can be goal-directed and policy-constrained. The self-management properties of autonomous systems, i. h Self-configuration, self-optimization, self-healing, and self-protection can be designed in terms of unsupervised learning and may involve replanning and reprogramming to achieve a goal when internal to the system and external environmental changes would otherwise. cause the system to fail to achieve its objectives. However, goal-based generic IS models may lack an explicit representation of how they can control their environment, i. h lack explicit control algorithms. Architectures are sometimes needed for UbiCom systems that can operate in multiple environments. For example, in the adaptive transportation planning scenario, the transportation system must take into account the human environment in terms of the needs of the people to be transported, the different departure times, and the number of people waiting at certain points for transportation. be picked up until. The transportation system must be aware of the physical environment of any physical obstacles on the route in order to predict the time of arrival at specific points along the route. Transport

Environment input events

Surroundings

Output events Input events Mediator

output events

physically reactive model

human environment model

human reactive model

Target model (autonomous)

human environment model

human reactive model

Target model (autonomous)

physically reactive model

Horizontal Layered Hybrid Model Design

Hybrid Vertical Layer Model Design

Figure 8.9 Hybrid IS designs to support UbiCom

Semantics KB IS

263

The system can be controlled by targets and supply functions to reach all pickup points on time and maximize the goods you pick up. Figure 8.9 shows two different hybrid designs that differ in two different dimensions. First, different IS models can be run in chain (vertical layer model) or in parallel (horizontal layer model). Second, for the chained model, the order in which the models are executed can greatly affect computational efficiency, which is a particular concern for systems with limited ICT resources. In Figure 8.9, the chained design is based on environmental events processed first in a human environment model, then in a goal-based model, then in a reactive human model, and finally in a reactive model of the physical world. For example, in the adaptive transportation planning system, the IS system first considers the world time-of-day model to schedule a larger capacity vehicle during peak hours. Next, it takes into account the goals of the transportation system in maximizing revenue by accommodating as many riders as possible. He then discovers that although passengers are waiting at a pickup point, it is already full, so he doesn't stop to pick up more passengers. Eventually, the transportation system notices that the road conditions are wet and, along with a full load, must brake earlier and slower to stop the vehicle if necessary.

8.4 KB IS Semantics There are a number of representations that can model knowledge in terms of concepts and their relationships. Expressed informally12 as a collection of descriptions of the world that help users define the meaning of their actions in the world, the term ontology is often used interchangeably with the term knowledge representation.

8.4.1 Knowledge representation According to Davis et al. (1993) a knowledge representation (KR) is best understood in terms of five distinct functions that it performs. First, a KR behaves like a surrogate, a surrogate for the thing itself, which is used to allow an entity to determine consequences, that is, by arguing about the world rather than acting in it. Second, a KR is a set of ontological commitments, in the sense that a concept is generally related to its relations to other concepts through use and is understood by its relations to other concepts. Third, a KR is often used for intelligent thinking. Fourth, a KR enables efficient computation of machine-readable13 and machine-understandable14 knowledge. Fifth, a KR provides a language for people to express things about the world. There is a range of ontological models and representations, depending on how concepts and their relationships are defined and organized. On the one hand, there are lightweight ontologies like dictionaries that have simple conceptualizations and contain parts like term values that may not be machine-readable and not machine-relatable to other terms.

Ontologies have much more formal definitions, i. h as formal and explicit specifications of a shared conceptualization (Nicola, 1998; Gómez-Pérez, 1999). 13 Machine-readable simply refers to data that can be read by a computer, and where reading implies that the system can understand the structure of the data, for example, to extract terms that match a query term. Machine readable may not be human readable and vice versa. 14 Machine understandable refers to data in which a computer can interpret the meaning of the data in terms of a (usually very limited) conceptual framework, such as a categorization of concepts. The machine understandable may not be humanly understandable and vice versa.

264

Intelligent Systems

Currently, the most widely used lightweight KRs appear to be those based on the W3C Web XML15 model such as the basic node-tagged graphical representation and extensions developed as part of the Semantic Web (SW). XML is a language designed to exchange extensible hierarchical data structures specific to applications and domains. This is used by many web services designed around a service-oriented computing architecture (Section 3.2.4). These data structures are certainly machine readable, but XML is a difficult data format on which to build machine understandable automated processing and to support interoperability between autonomous heterogeneous Internet services. At the other end of the scale are the heavier ontologies that support more descriptive conceptualizations and more expressive restrictions on the terms and their interrelationships, including logical restrictions (Corcho et al., 2003). Regardless of the specific ontology properties, heavier ontologies generally include the following elements: taxonomic relationships between classes (also called categories), data type properties, descriptions of class member attributes and object properties, relationship descriptions between class members, and to a lesser extent class instances and property instances. Although semantic heavy KRs and logical KRs may have a common representation that supports semantic queries and logical reasoning, the question arises as to what kind of logic should be combined with the semantic representation. When the use of the KR requires temporal thinking, uncertainty thinking, or clear logical reasoning, different logical representations are needed, and in some cases, multiple logics may be required to reason about knowledge. For this reason, in this text, the general non-logical conceptualization and relationships discussed in this section are separated from the use of knowledge to support different types of logical reasoning (Sections 8.4.2, 8.6). In terms of computational representations, KR can be displayed in graphical notation, also known as semantic network notation. There are many variations on these graphics to represent the different light to heavy KRs. An example of a light and a heavy is shown in Figure 8.10. The graphical notation can be reduced to an exact declarative form for computational purposes, such as B. Reasoning about concepts in the knowledge model. Different communities within computing have also developed similar overlapping models. For example, early versions of the KR type called frames developed by the AI community are quite similar to the object-oriented model developed by the software engineering community.16 According to Berners-Lee et al. (2001) and Shadbolt et al. (2006) initiated SW in the 1990s as a vision for the development of the web from machine readable (machines can read and parse syntax to extract concepts) to machine understandable (machines to act on the meaning of words). concepts) and around to support broader service interoperability between heterogeneous service processors called agents. SW defines a set of ontology language models based on RDF17 as an XML extension that supports rendering of graphs with edge labels, on RDFS18 and on OWL19 as an extension of RDFS that acts as a KR heavyweight and

quince

Extensible Markup Language (XML), see http://www.w3.org/TR/REC-xml/, accessed January 2007. XML defines an unnamed hierarchy of concepts and properties. 16 frames have slots that can represent other frames in the hierarchy or actions. When an event enters a frame, it is checked against the relevant slots, which can trigger slot updates and more frames. Software objects in an object-oriented model are similar and can be triggered by events as part of an ECA model. 17 The Resource Description Framework (RDF), see http://www.w3.org/TR/REC-rdf-syntax/, accessed January 2007. RDF adds support for named associations between concepts to XML. 18 RDF Schema, see http://www.w3.org/TR/rdf-schema/, accessed January 2007. RDFS adds support for category relationships to RDF. 19 The OWL Web Ontology Language, http://www.w3.org/TR/owl-ref/, accessed January 2007. OWL adds RDFS support for scope and domain constraints, existence and cardinality constraints, transitive, inverse and symmetric. properties and for logic.

Semantics KB IS

265

Node-labeled graph - uses some kind of inter-node binding and inter-node binding, e.g. B. W3C XML Device

MTOS device

ASOS Device

on-screen keyboard

static control panel

Computer

mobile phone, mobile phone

telephone

RTOS device

Display device AV recording device

Link Labeled Chart – uses various types of links, e.g. B. W3C RDF, RDF-S, OWL, etc. -an MTOS device ASOS device screen Has a panel Is-a Is-an Is-a RTOS device Mobile Static Keyboard Instance Is-an-instance-of-1 , n-Phone Owner AV Player Screen AV Recording Computer Phone Ben Device Device Device owned by a person

Figure 8.10 Two different graphical KRs for the device domain: a weaker, less expressive, node-labeled graphical representation and a stronger, edge-labeled graphical representation

Contains support for logical thinking. There is a conceptual model called the Semantic Web Layer model, presented by Berners-Lee in 2003, which describes how these different models are related (Horrocks et al., 2005). The same type of concepts will have a different structure or syntax in these different types of XML-based ontology models. They also differ in how explicit the semantic constraints and relationships are enriched.

8.4.2 Design issues There is still an ongoing debate about what should be specified in a KR model. Horrocks et al. (2005) consider how the KR model of the Semantic Web should support rules and how a closed world versus open world assumption should be modeled.

8.4.2.1 Open world versus closed world semantics RDBMS-type KB models tend to adopt closed world semantics; h if the data is not present, it is wrong (negation as error). By contrast, semantic-type KBs tend to use open-world semantics that treat absence as unknown (negation as unknown). Sometimes semantic KBs can be allocated to an RDBMS for more efficient and manageable storage. Because queries for the KB semantic type are not implemented as relational queries, this difference in open-world and closed-world semantics should not be a problem. Another problem is that the data semantics are often contained in the application code that creates the data storage structures and is not stored with the data. Application semantics are often not explicit and cannot be accessed from the data.

266

Intelligent Systems

8.4.2.2 Ciclo de vida del conocimiento y gestión del conocimiento La gestión basada en el conocimiento (KM) incluye un ciclo de vida de generación, suministro y mantenimiento del conocimiento.

8.4.2.3 Knowledge creation To create a useful and accurate application domain-based knowledge system requires a combination of problem understanding and a collection of problem-solving heuristics rules that have been shown to work well in the field. domain by experience. necessary. The older type of knowledge-based systems tended to use the knowledge of one or more human domain experts as the source of a system's problem-solving strategies and are often referred to as an expert system. For example, DENDRAL, developed in the late 1960s (Buchanan et al., 1969), was designed to infer the structure of organic molecules from their chemical formulas and from mass spectrum information about chemical bonding. present in the molecules. Many interactive knowledge generation tools are available today, allowing less experienced and specialized developers and users to create knowledge models and then export the knowledge model from the tool in a form that can be imported, interpreted, or analyzed. and then accessed through an API by computer applications. , see for example Prote'ge'20 or Jena21. The creation of an ontology is part of a knowledge representation process that is intelligible to machines but also to humans, based on a shared understanding of how humans represent, understand, and acquire knowledge, rather than how people do it. machines. For example, all objects that have a microprocessor, microcontroller, or central processing unit, have memory, have an input interface, have an output interface, have a network interface, and are collectively called "devices." The concept of device is unique in our ontology and has only one definition. A concept represents an idea of something, which can be a real world object or an abstract object like human behavior or feelings, etc. In general, all these concepts do not have an absolute definition; they are defined in relation to other concepts (ontological ties of the primary concept to the secondary concepts). In the example above, the terms "input interface", "output interface", "memory", and "microprocessor", "microcontroller", or "CPU" were used to define the term "device" (see Figure 8.10). Thus, concepts are understood through their relationship with other concepts that have already been understood and remembered. The process of creating a KR for a domain consists of the following steps:

• Definition of a taxonomy of concepts to specify categories in terms of generalization and specialization. • Define a set of relationships used between concepts and between concepts and properties of concepts. This set of relationships can be organized in a hierarchy.

• Defining constraints on the value in a relation and constraints on the values of concept properties, eg. B. positive integer, etc.

• Definition of axioms22 for relationships and concepts (for a logic-based KR).

Prote'ge', an open source ontology editor and knowledge base framework. Home page, http://protege.stanford.edu, accessed January 2007. 21 Jena, an open source semantic web framework for Java. Homepage http://jena.sourceforge.net/, accessed January 2007. 22 Where axioms are taken to be facts, propositions, and propositions (concepts connected by defined operators) that are always true, such as the proposition 'is- owner' is the opposite of 'belongs to'. However, there is also the question of whether or not this is true in selected domains, or in all domains, or in a possible world, or in all possible worlds.

Semantics KB IS

267

There are many different variations of this process; see, for example, Noy and McGuinness (2001). As with any type of design, ontology design involves making many modeling decisions. Some decisions have to do with categorization decisions, whether or not to model something as a subcategory of a category, as an instance of a category, or as a composition, e.g. for example, the "owner" of a device could be modeled as an instance of a "user" of a device, or a subcategory of users. There are ways to model relationships with different cardinality, such as B. One-to-one, one-to-many, or many-to-many. Decisions must be made regarding the direction of relationships and whether or not two directed relationships are opposites, e.g. for example, the relation "is_owner_of" is the inverse of the relation "belongs_to". There are options for which constraints can be used on relationships between properties and between categories. The ease and clarity with which complex relationships can be modeled also depends on the type of ontological models, such as B. light or heavy. Often, a knowledge model created requires a process or refinement to improve and validate it, perhaps by using the KB model to solve sample problems that allow stakeholders to critique its behavior and make necessary changes or modifications to the knowledge of the model. Program. This process is repeated until the program has reached the desired level of performance.

8.4.2.4 Implementation and maintenance of knowledge The easiest knowledge model to implement and maintain is a single domain or world model that has gone through a refinement process involving all stakeholders. This knowledge model is understood and agreed upon by them prior to implementation and then remains unchanged during implementation. In practice, however, knowledge delivery is often complicated by the existence of autonomous interest groups developing overlapping but different ontology models. Also, ontological commits can often only be derived via deployment, which can change the commit ontology model. This is because the understanding of the domain and the applications of a domain varies over time and in relation to different stakeholders, leading to changes in the domain and changes in conceptualization. Therefore, there are often several possible semantic models within a domain and they have to be related. The use and interoperation between heterogeneous ontological models during the conceptual world modeling process can be broadly classified into two types: fusion or integration and alignment. Noy and Musen (2000) define view merging as the creation of a single coherent ontology containing information from all sources, and alignment as a process in which sources must be consistent and coherent but kept separate. This may involve maintaining local ontology containers for each data source, resulting in a multilateral ontology model. The fusion approach often leads to the creation of a global knowledge model in which individual local ontologies can be mapped to each other. The alignment approach avoids the process of creating a global knowledge model, but instead maps specific semantic content directly between local ontologies (Poslad and Zuo, 2008). Ontologies within a domain are heterogeneous, not only because independent knowledge domains can be modeled differently, but also because they are subject to different ontological constraints in different application domains: the pragmatics or particular context of the use of concepts will vary. . Different human users may use different subsets of concepts with different semantics based on their understanding. It can be useful if the knowledge base tailors its responses to queries to the context of the users (Poslad and Zuo, 2008).

8.4.2.5 Design issues for using UbiCom Metadata, data that describes data and often summarizes data, is entered manually or automatically, e.g. B. by summary, derived from data that can be used to annotate and annotate data

268

Intelligent Systems

improve the search for information. This is because searches can be performed more efficiently by testing a subset of the metadata to match the data queries, rather than testing the entire data. If the metadata also uses a representation with well-defined semantics, semantic searches can be performed on the metadata in addition to syntactic searches. The computing resources of some devices, especially mobile, ASOS and embedded devices, are limited by design. It may be that the semantic information and commands cannot be processed well on the device. The amount of computing time is a resource that crucially affects the semantics of its results, since contexts are more likely to change in dynamic environments and when resource-constrained systems are in dynamic environments (Kaelbling, 1991). Care is required that the intended semantics of the output of a computation are valid given the time it takes to perform the computation and the computational resources are appropriate to use the semantics. It is necessary to choose designs for IS operations that match the ICT resources of the devices, e.g. For example, the trade-offs between small memory and longer computation must be carefully considered. Devices embedded in a local environment often have only a partial view, rather than a global view, of their environment. This may be because the systems do not have sufficient resources, such as the time and capacity to acquire and access the required global knowledge. Today's systems often generate information in a much more decentralized way than in the past. They are not isolated systems, but often bring together large communities in ad hoc situations. This social dimension (of networking), as found in social networks, creates environments for partial and dynamic information. AI approaches can help manage incomplete and uncertain information and support the decision-making process, either autonomously or through human intervention.

8.5 Classical Logic SI Classical logic, based on first-order logic,23 is considered by many researchers to be the heart of any SI that should support system closure. These types of IS are also commonly known as (logical) reasoning systems, deliberative IS, and symbolic AI, as these systems involve the manipulation of symbols in the form of logical formulas, although symbols in general can also refer to any mathematical formula. , including algebraic formulas.

8.5.1 Propositional and first-order logic Propositional logic is the simplest logical model in the world. Here knowledge is presented in the form of sentences, statements or relationships that are true or false. Multiple declarations can be combined with logical operators or connectors for literals that represent the things in the KB that you are working on. The standard logical connectors are: and (also called a conjunction, also denoted A ^ B, A \ B), or (also called a disjunction, A _ B, A [ B), not (also called a negation, :A), equals and implies (as in A B implies A) expresses B if A then governs B). These connectors can be used to form sentences, which are well-defined formulas that follow a defined structure or syntax. The meaning or semantics of these operators can be described in truth tables.

True or false logic is also known as clear logic and binary logic, whose states can be represented as one or zero. It is also the same logic used by the underlying digital electronic hardware used by ICT systems.

The classical logic is

269

In first-order logic,24 predicates are defined to support more meaningful propositions than statements that allow a property to be related to an object or a property to a value. The sentence "device A is in the idle state" expresses a predicate, which can be thought of as a type of function, also known as propositional functions, that is applied to individuals (which are generally not propositions) and produces a proposition, B . Mode (device A, idle state). The main difference between propositional and predicate logic is that in propositional logic all relations are either true or false whereas in predicate logic predicate functions are neither true nor false even though their evaluation may be. In addition to expressing individual relationships, more general relationships can be expressed using two additional logical operators, more specifically called quantifiers. The universal quantifier (for all) expresses the idea that all properties have a specific value, while the existential quantifier (exists) expresses the idea that at least one property has a specific value.

8.5.2 Conclusion Conclusions, also called conclusions, include logical operations on logical sentences or statements within a (logical) model supported by the model, e.g. the world, a domain of application, etc., to draw conclusions and derive other propositions, e.g. B. A includes B, A |¼ B. Inference is used to find consequences. Sometimes several possible worlds or models will be possible because the SI has an incomplete model of the world, because the SI is not sure which model will come true in the future, and perhaps because different viewers and applications may also have their own model of the world. world. world (Section 8.4.2.4). Pattern checking is used to check whether the conclusions of propositions are valid in all possible worlds or models. Valid sentences are called tautologies. Sometimes it is only necessary to check whether a proposition is true in a given model, ie h is satisfied in this model, instead of being able to say that it is true in all models, ie h is valid in all models. Pattern verification may include a change in logical restructuring, a change in the syntax of a logical sentence with the same semantics, to facilitate the verification of the logical equivalence of two sentences. Various laws of logic are used to assist in model verification, e.g. B. commutative laws that state A ^ B ¼ B ^ A, associative laws that state (A ^ B) ^ C ¼ A ^ (B ^ C), etc. Model verification is used every time new knowledge is added. If only logical sequences are added, the model is said to be monotone. There are several standard rules of inference in propositional logic. For example, the modus ponens rule, which says that given the propositions A) B and A, B can be derived. An inference algorithm used for knowledge discovery may use one or more inference rules. If the inference algorithm finds only subsequent sentences, it is said to be true. Resolution-based inference attempts to show that propositions can be implied by proving that the negated proposition in this KB is unsatisfactory by proving that there are logical contradictions. Solving type inference can result in lengthy computations. If the logical sentences in the KB are restricted to Horn sentences, which is a disjunction of literals where at most one is positive, then z can be used and forward chaining is linear as the size of the KB increases, i.e. h becomes it requires much less computation than solving (Russell and Norvig, 2003, pp. 194-239).

The most common form of predicate logic is called first-order logic, or FOPL. Predicate logic is also known as first-order logic, FOL, or FOPL, or first-order predicate calculus, or FOPC.

270

Intelligent Systems

Predicate logic in KB can be transformed into propositional logic by using rules to instantiate universal and existential quantifying propositions so that propositional logical inference can be used. However, the propositional logic approach is usually quite inefficient for predicate inference, since we have to derive every instance of a sentence that contains a universal quantifier. First, the efficiency of propositional reasoning can be improved by removing first-order rules of inference, such as modus ponens, so that they can be applied more generally. Second, substitutions can be found and used to make logical sentences appear semantically equivalent in a process called union. Specific FOL programming languages, the best known being PROLOG, PROgramming in LOGic (Bratko, 2000), tend to support FOL Horn Clause and tend to interleave logic with control or instructions for working with logic. PROLOG enables the development of applications and tools that can search logical sets stored in a KB using forward and backward chaining search algorithms. It can also support programming and machine learning. Automatic reasoners, also called theorem provers, have also been developed. These tend to use full FOL, unlike logic programming languages, and tend to further separate control logic from logic representation itself. A variety of problems can be approached by presenting the problem description and relevant background information as logical axioms and treating the problem instances as theorems to be proved. This idea is the basis for work on automatic proof of theorems and systems of mathematical reasoning. Many important problems, such as the design and verification of logic circuits, the verification of the correctness of computer programs, and the control of complex systems, seem to respond to this approach.

8.5.3 Design issues Of all types of logic, first-order logic has the most consistent set of axioms, making the integrity of the logic model more scalable25 for use by many systems compared to, for example , computer science and relational databases. Soft computing (Section 8.6) relates the truth of something to the amount of evidence and the accuracy of the underlying concepts. This may vary depending on the context of use, including time. KR-type relational databases (RDBs) (Section 12.2.9.5) use relational algebra to allow knowledge organized as data based on sets or tables to be queried to create subsets or form supersets. However, RDBs operate on the closed world assumption and make assumptions about whether something is true based on its existence in the database. If it does not exist in the database, it is considered incorrect. Thus, different databases within a domain may disagree about what is fundamentally true or false in a domain because they contain different records for the same domain. Therefore, many KR systems are often based on classical logic. However, the main limitations of classical logic must also be noted, such as the difficulty of expressing exceptions, imprecision, uncertainty, and the degree of computation sometimes required to establish the truth of complex logical structures. In practice, logical inconsistencies can occur in a distributed knowledge base unless each new entry is checked against the full knowledge base and new logical propositions that make the model non-monotone are not avoided. There are also many different subtypes and extensions of classical logic. Sometimes the reasoning may require the use of multiple logics, e.g. B. Time and uncertainty. It is not at all clear how to verify the completeness and robustness of multiple types of logic models. Logic-based IS may not work efficiently in time-constrained environments. In logic-based KB IS, perceptions are likely to be symbolic. But for

Scalable in this sense means that logical conclusions and axioms yield consistent results, such as truth values, across a variety of potentially heterogeneous sources of knowledge.

Soft Computing IS-Modelo

271

In many settings, it is not obvious how mapping from environment to symbolic cognition (eg, images) could be done. It is not clear if a single logic will suffice or how hybrid logics can be used. Representing the properties of real-world dynamic environments (for example, temporal information) is challenging.

8.6 Soft Computing IS Models Many decisions involving interactions with humans and the physical world are soft and are not expressed as true or false. These are quite qualitative and may be associated with some inaccuracy and uncertainty. Such systems can be designed using soft computing techniques, e.g. B. Probability theory to work problems with uncertain states and fuzzy logic to work problems with imprecise states.

8.6.1 Probabilistic networks There are several notions necessary to model the probability of occurrence of indeterminate events or to model the degree of belief and reasoning about a proposition or predicate. There is a prior or unconditional probability of a statement or event in the absence of information. When the information is known, conditional or posterior probabilities, expressed as the probability of A, should be used since we know the probability of B, expressed as P(A|B) ¼ P(A ^ B)/P(B). A conditional probability is expressed by two unconditional probabilities. The probabilities for the individual values of a statement are defined in a probability distribution either as a set or vector of discrete functions or as a probability density function. Multiple probability distributions for different statements can be combined with common probability distributions, e.g. B. P(A^B), the joint probability of weather and traffic conditions. A complete associated probability refers to the entire set of different event probabilities that are associated. Using the product law P(A ^ B)¼P(A|B) P(B) and commutativity such that P(A ^ B) ¼ P(B|A) P(A) P(B|A) ¼ ( P (A|B) P(B)) / P(A) to derive what is called Bayes' rule or law or theorem. This is the basis of probabilistic inference. This expresses a conditional probability in terms of another conditional probability and two unconditional probabilities. A Bayesian Network (BN), also called a belief network or probability network, can be used to represent any complete joint probability distribution. A Bayesian network can be used to reason in a context-aware UbiCom scenario where nondeterministic preconditions and nondeterministic outcomes exist. For example, in the adaptive vehicle planning scenario, both passengers and buses can arrive at pickup points indefinitely, and people and vehicles can wait indefinitely (Figure 8.11). The scheduled vehicle arrives at the pick-up point (Va)

The vehicle waits at the pick-up point until the passengers arrive (Vw)

P(Va): 0.98

P(Ha): .70 people arrive at the collection point (Ha) Va: Ha P(Sp) vehicle arrives at t: t 0.85 collection point and stops until t: f: 0.9 picks up passengers (Sp) f: t : 0.8 f : f: 0.1 Sp: P(Vw) Sp: P(Hw) People wait at the collection point t: 0.05 t: 0.9 until the vehicle arrives (Hw) f: 0 .9 f: 0.5

Figure 8.11 A Bayesian network modeling vehicles and passengers arriving and waiting at pickup points indefinitely

272

Intelligent Systems

8.6.2 Fuzzy logic Fuzzy logic is useful for representing a model where the result of a statement is deterministic but somewhat approximate or imprecise, such as B. when the vehicle is driven very slowly, or slowly, or at a moderate speed, or fast, or very fast. This type of imprecision can also be used in fuzzy rules. For example, in the adaptive transportation planning scenario, a fuzzy logic rule might be if the bus is slowly moving away from the pickup point and a passenger is moving quickly toward the pickup point, then slow the vehicle down to stop near the pickup point. . Here the terms "slow", "fast" and "almost" act as confusing descriptors. Conversely, a clear logic rule for the same situation in the scenario would not be able to distinguish between the bus that is moving slowly away from the pickup point and may stop near the pickup point and a bus that is moving quickly away from the pickup point. pickup point and must stop far from the pickup point if you determine that additional passengers can be accommodated.

8.7 IS System Operations The operations of the generic IS model, including searching and scheduling, are discussed here. Reasoning as an IS operation is intertwined with the use of specific logic as a representation for the IS model, which is covered in the Crisp Logic and Soft Computing sections. Learning is also a generic IS operation, but it requires specific architectural support for feedback and is therefore covered in the Learning IS section.

8.7.1 Search Search is a problem-solving technique that systematically explores a problem state space, i. h Sequential and alternative steps in the problem solving process to select a goal state or a chain or path through intermediate states to reach a goal state. This space of alternative solutions can then search for an answer (Newell and Simon, 1976). Much of the initial research for State Space Search was conducted using common board games such as checkers, chess, and 15 puzzles. In addition to their inherent intellectual appeal, board games have certain qualities that made them ideal subjects for this initial work. Most games are played according to a clearly defined set of rules. This facilitates the generation of the search space and frees the researcher from many of the ambiguities and complexities inherent in less structured problems. The board configurations used to play these games are easily represented on a computer and do not require the complex formalism required to capture the semantic complexities of more complex problem domains. Examples of search problem states include: searching for alternate board configurations in a game to win the game (goal state); Search a set of annotated images of faces to see if an unknown face matches an already annotated one (in the personal memories scenario); search street intersections to find a route between the current location and a destination location (in the adaptive transportation planning scenario); Find a route through obstacles to guide a robot from a location to a destination. Thus, the problem is expressed as an initial state, a target state, an objective test function to test whether each state is the target state, and a utility function that maps a path between two states to some performance or cost metric. which can be added. to calculate the cost of achieving the target state. The basic type of search algorithms is to use an uninformed search, also known as a brute force search or blind search, where no clues are available on how to reduce the search space, e.g. B. Breadth-first search and depth-first search. Often the problem can be the search space.

IS system operation

273

2 5

breadth search

3 6

a search

30+200

20+300

10 depth search

5 100+160 5

2 3

8 7

150+80

4 5+400

6 50+170 9 120+100

9 100+70

Figure 8.12 Two types of uninformed or brute force search, a breadth-first versus depth-first search, and one type of informed search, the A* search

is displayed graphically (Figure 8.12). The search involves traversing the graph either bread first or depth first and testing each node to see if it is the target state. Unreported problem space searches tend to work in a forward direction (progression) from the initial state to the final goal state (called forward chaining). In some cases, the search algorithm must also handle the case that the IS system cannot identify its initial state, e.g. B. when a transportation vehicle is lost and needs a route to a destination. The mode of the search space may not be uniform, i. h different nodes can have different numbers of branches. Furthermore, the problem spaces should not be monovalent, but multivalent. Some problems, such as games, can generate extremely large search spaces that require large amounts of computation for unreported search techniques, e.g. B. Depth search can fail in infinite depth spaces, and where loops exist, time complexity (the number of nodes accessed is high), space complexity (the maximum number of nodes in memory is linear) and optimization (you may not always find a most cost-effective solution). For this reason, variations of this reported search can be employed, such as depth-limit search (depth-first search with depth limit) or an iterative deepening search (Russell & Norvig, 2003, pp. 59-93). A general solution to reduce the computation of an uninformed search is to use informed search techniques that use problem-specific information to limit the problem space. A central component of informed search is a heuristic function, defined in its most general form as a function that depends on the current node in a problem space, e.g. B. a cost function that returns a value to reach this current node from a previous node. When the nodes represent physical locations, the cost function could be related to the distance between nodes, the time it takes to travel between nodes, or the cost of power when traveling between nodes. The heuristic function that assigns a value to each node depends on the information about the problem. A variation of the cost function is to assign a first cost to get to the current node from the previous link and a second cost to get from the current node to the destination node. This cost heuristic is used by A*search or A-star, one of the most popular types of informed search or search heuristics (Figure 8.12). A central application for search in general is information retrieval. The point here is to reduce costs in terms of the number of tests targeted for each node. This is typically accomplished by testing indexes or metadata rather than using actual data in searches and by using efficient index structures such as sparse indexes, B-trees, hashing, linked lists, and R-trees to support informed searches (Watson, 2006 ). , p. 326–341).

274

Intelligent Systems

8.7.2 Classical (deterministic) scheduling Much of the early scheduling research began with attempts to design robots that could perform their tasks with some degree of flexibility and responsiveness in the physical world. Planning assumes that a robot is capable of performing specific atomic actions. Try to find a sequence of these actions that accomplishes some general task, such as B. Moving through a room full of obstacles. Today, research in planning extends well beyond the fields of robotics and includes the coordination of complex tasks and objectives. Modern schedulers can be used both for robots or embedded software agents, and for complex adaptive controls in machines such as particle beam accelerators. Planning involves finding a plan and then executing it. Searching for a plan uses: a representation of the planning model; backward chaining (from the target state to the current state) to determine the action chains leading to the target state, forward chaining to reach the target state from the current state; heuristic-based informed search techniques (Section 8.7.1), since otherwise uninformed searches would be too inefficient; Decomposition of a complex problem into simpler problems that are easier to solve (for example, divide and conquer). The planning model represents states, goals, actions that transform states into goals, chains of actions between non-neighboring states, and heuristic cost functions. It also allows to restrict the selection of multiple action strings using some heuristics, e.g. For example, a transportation system that needs to accommodate multiple passengers at multiple locations may be constrained by travel between locations to minimize fuel consumption or time required, or a combination of both. This can be modeled as a graph where the nodes represent states and the connections between nodes represent actions. In Figure 8.12, the links representing the actions are not labeled to identify the actions, but this information could easily be added to form a labeled graph at both the edge and the node. Actions can be represented in terms of preconditions, which, when true, allow actions to be triggered, postconditions or effects, which define what should now be true if an action is successful.26 As mentioned above, an important technique To make complex planning problems more solvable, the problem must be decomposed to reduce the problem space to search for. Search techniques often do not take into account the decomposition of the problem into subproblems, computing forward or backward strings for subproblems and then combining them to form a solution to the overall problem, resulting in linear programming rather than linear programming. exponential programming. programming algorithms. A well-known problem decomposition method is based on hierarchical task analysis, or HTA, in which a goal is solved by performing a set of high-level actions, which are then refined to lower-level actions. In later steps these can be further refined until only primitive actions remain that cannot be further refined (Figure 8.13). Plans represent the decomposition of a high-level action into a series of lower-level actions. Plans are stored in a plan library, from which they are selected to form a path to the objectives. A limitation of forward and backward state lookups is that they depend on the order of each current state evaluated relative to a path to the target state. They also cannot take advantage of actions that can be scheduled in any order to achieve a goal, as long as they do not interfere or interact with each other (partial order planning, or POP in Figure 8.13). In some cases, a chain of actions, which can occur in any order, can achieve the goal. POP can reduce the size

Event-based models can also be used to define events and action preconditions, but they tend not to define action postconditions or define an explicit action representation.

IS system operation

275

0 View AV content

Plan: Do (1),2,(3),4 ; Actions 1.3 are optional

1 Setup Screen 2.1 Set Record Source

2.2 Link screen to player

2.1.1Import source to player

Plan: Do 1, 2, 3 in any order, then watch 4 AV content

3 setting environment

set up 2 players

4.1 Playback 3.1 Adjust sound

3.1.1 Close window settings 1. View 2 player settings

4 interactive playback

3.2 Lighting adjustment 3.2.1 Cover window

4.2 Pause

4.4 Rewind 4.3 Fast Forward

3.2.2 Dim Ambient Light

partial order plan

4 interactive playback

3 Configuration environment Figure 8.13 Hierarchical task plan and subtask plan for "View AV content" goal.

looking to find an action plan and gives the plan additional flexibility, e.g. B. when certain actions are temporarily delayed. In goal-oriented IS design, planning serves to select a goal-oriented chain of action. Therefore, every action taken is part of the plan. In some types of system interaction, environmental events can trigger an action for which there is no current plan or goal, a situated action. The situated action would trigger a goal-based IS design to create a plan for it.

8.7.3 Non-deterministic scheduling Classical scheduling considers scheduling in static, observable, and deterministic environments with systems that can act correctly and whose plans are fully executed. However, some environments can be non-deterministic and partially observable, and this is where classical planning fails because it cannot determine a course of action. There are two types of planning methods, sensorless or conformal planning and conditional planning, which can work in environments of limited uncertainty where actions have unpredictable effects but where the set of all possible assumptions and effects is fixed. For example, in contingency planning or conditional planning, plans are created with different branches representing different conditions. During plan execution, the agent fetches the state and then selects the appropriate contingency branch. Contingency planning requires the ability to recognize actions as they occur and take advantage of a limited number of contingencies. An example of conditional programming is used in context processing (section 7.2.4). There are also two types of planning methods, monitoring and replanning and continuous planning, which can operate in infinite uncertainty and where the set of assumptions and effects is unknown or very large (Russell and Norvig, 2003, pp. 417-461). .

276

Intelligent Systems

EXERCISES 1. Compare and contrast the types of architectural models for the IS given in Section 8.3, i.e. h reagent, environment model, objectives, tools and learning, in terms of how actions are selected, how the model is represented and what types of environments, e.g. , deterministic, episodic, etc. these models are suitable. 2. Consider the design of a so-called vertical layer model. Is this a good design if the reasoning layers are activated before the reactive layers or vice versa? Compare and contrast different IS model arrangements within a hybrid IS model for different application scenarios. 3. Discuss what kind of expressiveness, ranging from ontology-light to ontology-heavy, is needed in knowledge modeling used by reactive, environmental model-based, and goal-based IS designs. 4. Describe the need to make a KR machine readable and understandable. 5. Present the design of a system consisting of a set of devices with limited resources that support machine-readable and understandable data. 6. What are the differences, if any, in the operation of an automatic reasoner compared to the use of a logical programming language? 7. What is first order logic? Explain the difference between this and propositional logic. 8. Explain why first order logic (FOL) is so prevalent in KB systems. 9. What is soft computing? What is the difference between inaccuracy and uncertainty? Discuss some specific model types for modeling model inaccuracy and uncertainty. 10. Show how uncertainty in planning can be managed.

References Berners-Lee, T., Hendler, J. & Lassila, O. (2001) The Semantic Web. Scientific American, 284(5): 34-43. Bratko, I. (2000) Preface Programming for Artificial Intelligence, 3rd Edition Harlow: Pearson Education. Buchanan, BG, Sutherland, GL and Feigenbaum, EA (1969) DENDRAL heuristics: a program for generating explanatory hypotheses. In B. Meltzer and D. Michie (eds.) Machine Intelligence, vol. 4. Edinburgh: Edinburgh University Press, pp. 209-254. Coppin, B. (2004) Artificial Intelligence Enlightened. Sudbury, MA: Jones and Bartlett Publishers. Corcho, O., Ferna´ndez-Lo´pez, M. and Go´mez-Pe´rez, A. (2003) Methodologies, tools and languages for the construction of ontologies: where are they found? Data and Knowledge Engineering, 46(1): 41-64. Davis, R., Shrobe, H. & Szolovits, P. (1993) What is a knowledge representation? AI Magazine 14(1):17–23. Franklin, S. and Graesser, A. (1996) Is it an agent or just a program?: A taxonomy for autonomous agents. In Proceedings of the Workshop on Intelligent Agents III, Theories, Architectures and Languages of Agents. Lecture Notes on Computer Science, 1193: 21–35. Frege, G. (1879) Conceptual notation: a formulaic language of pure thought modeled on the formulaic language of arithmetic. Halle: L. Nebert, 1879. Trans. S. Bauer-Mengelberg, in Jean Van Heijenoort (ed.) (1967) From Frege to Godel: A Source Book in Mathematical Logic, 1879–1931. Cambridge, MA: Harvard University Press. Friedman-Hill, E. (2003) Jess in action: Java rule-based systems. Upper Saddle River, NJ: Pearson. Go'mez-Perez, A. (1999) Ontological engineering: a state of the art. Expert update. Han, J. and Kamber M. (2006) Data Mining: Concepts and Techniques, 2nd Edition San Francisco: Morgan Kaufmann Verlag. Horrocks, I., Parsia, B., Patel-Schneider, P., et al. (2005) Architecture of the Semantic Web: Stacks or Two Towers? In Principles and Practice of Semantic Web Reasoning (PPSWR 2005), pp. 37–41.

references

277

Jasper, R. and Uschold, M. (1999) A framework for understanding and classifying ontology applications. In Proceedings of the IJCAI-99 Workshop on Ontologies and Problem Solving Methods: Lessons Learned and Future Trends, Stockholm, Sweden, August 1999. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/ Vol-18/11-ushold.pdf Kaelbling, L.P. (1991) A situated automaton approach to embedded agent design. ACM SIGART Bulletin, 2(4): 85-88. Karray F. and De Silva C. (2004) Soft Computing and Intelligent Systems: Design: Theory, Tools and Applications. Upper Saddle River, NJ: Pearson Books. Lindwer, M., Marcelescu, D., Basten, T., et al. (2003) Visions and achievements of ambient intelligence: linking abstract ideas with real concepts. In Design, Automation and Test in Europe Conference and Exhibition, pp. 10-15. McCarthy, J. and Hayes, PJ (1987) Some philosophical problems from the point of view of artificial intelligence: readings in non-monotonic reasoning San Francisco: Morgan Kaufmann, pp. Mitchell, TM (1997) Machine Learning. New York: McGraw-Hill. Newell, A. and Simon, HA (1976) Computer Science as Empirical Research. ACM Notices, 19:113-126. Nicola, G. (1997) Understanding, construction and use of ontologies. International Journal of Human and Computer Studies, 46 (2–3): 293–310. Nicola, G. (1998) Formal ontology in information systems. In Proceedings of the International Conference on Formal Ontology in Information Systems, FOIS'98, Amsterdam, IOS Press, pp. 3-15. Noy NF and Klein M (2003) Ontology development: not the same as schema development knowledge. Information Systems, 6(4): 428-440. Noy, N.F. and McGuinness, D.L. (2001) Ontology Development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory Technical Report KSL-01-05. Available at http://protege.stanford.edu/publications/ontology_development/ontology101-noy-mcguinness.html, accessed January 2007. Noy, N. and Musen, M. (2000) PROMPT: Algorithm and tool for automated ontology merging and alignment. In Proceedings of the 17th National Conf. on Artificial Intelligence (AAAI'00), pp. 450–455. Nwana, H.S. (1996) Software agents: an overview. Knowledge Engineering Review, 11(3): 205-244. Poslad, S. and Zuo, L. (2008) An adaptive semantic framework to support multiple user views across multiple databases. In M Wallace, M Angelides, and P Mylonas (eds.) Advances in Semantic Media Adaptation and Personalization, Series: Studies in Computational Intelligence, Vol. 93. Berlin: Springer Verlag, pp. 261-284. Ramos, C., Augusto, J.C. and Daniel, S. (2008) Ambient intelligence: the next step for artificial intelligence. IEEE Intelligent Systems, 23(2): 15-18. Riley, J.R., Greggers, U., Smith, A.D. et al. (2005) The flight paths of honey bees recruited by the wobble dance. Nature, 435 (May 12): 205–207. Russell, S. and Norvig, P. (2003) Artificial Intelligence: A Modern Approach, 2nd Edition Upper Saddle River, NJ: Prentice Hall. Shadbolt N, Berners-Lee T, and Hall W (2006) The Semantic Web Revised. IEEE Intelligent Systems, 21(3):96-101. Suchman L.A. (1987) Plans and situated actions: The problem of man-machine communication. Cambridge: Cambridge University Press. Tarski, A. (1944) The semantic concept of truth and the foundations of semantics. Philosophy and Phenomenological Research, 4:341–375. Available at http://www.ditext.com/tarski/tarski.html. Thompson, SG and Azvine, B. (2004) There is no pervasive computing without intelligent systems. BT Technology Magazine, (22) 3: 39-49. Watson, R. T. (2006) Data Management: Databases and Organizations, 5th Edition Chichester: John Wiley & Sons, Ltd. Wielinga, B.J., Schreiber, A.Th. and Breuker, J. A. (1992) KADS: a modeling approach to knowledge engineering. Knowledge Acquisition Journal, 4(1):5-53. Special edition “The KADS approach to knowledge engineering”. Reprinted in: Buchanan, B. & Wilkins, D. (eds.) (1992) Readings on Knowledge Acquisition and Learning, San Mateo, CA: Morgan Kaufmann, pp. 92–116. Wooldridge, M. (2001) Introduction to multiagent systems. Chichester: John Wiley & Sons, Ltd. Wooldridge, M. and Jennings, N. (1995) Intelligent Agents: Theory and Practice. Knowledge Engineering Review, 10(2):115-152.

9 Smart System Interaction with Patricia Charlton

9.1 Introduction As more and more UbiCom devices improve and become more universally connected, we finally have the building blocks to enable intelligent interaction. Intelligent interaction refers to richer interaction beyond the use of basic universal network communication protocols, e.g. for example, TCP/IP. For example, intelligent interaction can use coordination to reduce the need for explicit communication when different autonomous systems work together, and can use semantics not only to share information and tasks, but also to share intentions, goals, plans, knowledge, and experiences. . In general, the ability to communicate in a meaningful way using rich language is often seen as a clear sign of intelligence.

9.1.1 Chapter Summary The diversity of interaction and attempts to compare and contrast CCI and HHI are described below (Section 9.2). Subsequently, intelligent interactive system designs are discussed, which are based on two different strategies, making system interaction more intelligent and making individual intelligent systems interact, i. h Use of MAS (Section 9.3.3). This is followed by a discussion on the specification of interactions (protocols) using network protocols, semantic exchange, and speech acts. The types of multi-agent system designs that support this last type of interaction are discussed. This chapter concludes with a brief overview of some intelligent system interaction applications (Section 9.4).

9.2 Multiplicity of interaction At its simplest level, interaction can be viewed as a multiplicity or cardinality of communication relationships between independent senders and independent receivers. The multiplicity of interactions can lead to resource contention by multiple users. Multiple transmissions cause conflicts and interruptions and can cause an overload of resources, such as power, due to redundant communication. The multiplicity of interaction also increases the complexity of the interaction and reduces the

Stefan Poslad

Smart system interaction

280

Determinism and Interaction Control. Therefore, the interaction needs some kind of control mechanism to prevent or detect and correct these problems. The mechanisms used for control depend on whether the interacting entities are dumb, smart but cooperative, or smart but competitive, i. h are selfish. There are several important types of multiplicity of interaction (Figure 9.1). This section discusses these types of multiplicity and how multiplicity can be modeled and managed in the different components of ICT systems (Table 9.1). A powerful model for designing this range of types of organizational multiplicity is that of multiagent systems or MAS (Russell and Norvig, 2003; Wooldridge, 2001; Fasli, 2007).

multiple answers

Multiple compositions 1 input Inputs from multiple sources/multiple transmitters

depending on the context of the recipient

Concurrency control for multiple requesters

Offerer

Mediator applicants acting as filters

Figure 9.1 Some examples of intelligent interaction: service composition, concurrency control for shared resources, contextual recipient responses, and active mediators acting as filters

Table 9.1 Summary of types of multiplicity and associated designs Main types of multiplicity of interaction

Resolving the multiplicity of interaction in different perspectives (V) of systems: human (H), physical world (P), services (S) and network (N).

Multiple independent transmitters and receivers

HV and SV: multimodal sensory interaction; Bidders, sellers participate in single or multiple markets NV: MIMO, e.g. eg PSDN, Mesh routing AV: RDBMS support ACID NV: IP packet switching NV: MAC, e.g. B. FDMA, TDMA, CDMA; Token Ring HV: personal assistants, intermediaries, etc. as agents SV: proxies, broker models, application caches NV: routers, wireless base stations HV: natural language such as (eg, speech acts) conversations SV: service interaction patterns, e.g. B. Delegation of tasks, response to requests, etc. HV, SV: Delegation of tasks, work processes, social cooperation versus economic competition PV, SV: Self-Star Systems

Use of common passive mediators Use of common active mediators Shared conversations Organized interactors and interactions Self-directed (self-organization) Semantic and mentalistic communication

HV, SV: common language, meaning, intentions, beliefs, etc.

variety of interactions

281

9.2.1 P2P interaction between multiple senders and receivers Multiplicity of interaction (or interaction cardinality) arises when people, tasks and devices that are distributed in space and time are linked to each other and to a larger part of the physical world it is augmented by digital devices (Figure 9.2). UbiCom can be designed and managed to support basic interactions between multiple independent transmitters and receivers, often referred to as Multiple Input Multiple Output (MIMO) systems. Communication is complex, but it can be handled by lower-level network protocols or by higher-level middleware services and hidden from applications. It may involve variable paths, delays, ordering, and attenuation or filtering. The same transmission can propagate along variable paths and act as passive intermediaries between senders and receivers. Transmission paths can be modified to accommodate spatio-temporal bottlenecks and interruptions (variable path). Data sent within the same session can take different paths through the underlying network system, resulting in different delays. Data may be received out of order (variable order) or the session may be terminated because it is not clear whether it will continue to reach the recipient. A single source can spread out a signal, causing different parts of a transmission to spread out and attenuate or reflect in different ways. It may not be possible to determine to what extent a filter has changed the original source (variable attenuation). Interaction diversity increases in operational and management complexity when multiple senders communicate with multiple receivers that are time-synchronized (multiple communications coincide in time) and when they are spatially synchronized (they are within the same spatial region). Multiplicity of interactions can include multiple senders (one-to-one-to-many), multicasts (one-to-many), multiple receivers, senders and receivers that are hidden, and using too few or too many. senders or include recipients.

9.2.1.1 Unknown Sender and Malicious Senders Recipients may not always be able to positively determine the sender ID or context. This can cause the following problems. A malicious sender can spoof someone else's credentials (sender masquerade). Malicious senders can send messages that, when opened by a recipient, can trigger malicious code. An observed error can propagate away from the error source and appear to originate from multiple possible error sources, but it may be important to determine the source of the error in order to handle it correctly. There are several strategies here. Messages from unknown senders may be kept in some quarantine areas and scanned for side effects.

M Receptor C B A

Multiple Unicasts

receptor M

Single recipient or very few recipients

? A ?

sender no.

1 sender

Received from unknown sender A, B, C or ?

transmission

Multicast or multiple unicast B

unknown recipients

too many senders

1 sender

Figure 9.2 Some basic examples of multiplicity of interaction

M relenting

282

Smart system interaction

Messages can be tagged by the broadcast middleware, but they must handle the case that the sender e.g. B. Sensors in sensor networks do not necessarily need to be clearly or incorrectly identified.

9.2.1.2 Unknown recipients Multicast (or broadcast) message transmission can be used by a sender to interact with unknown recipients, either to exchange information or to request a response. Multicasts to unknown recipients are widely used for service discovery in dynamic open systems and especially in P2P interaction. In a directory service or search engine, a requester often wants to be linked to any recipient or the first recipient they can match. There are several approaches to vendor selection by a third-party search engine or directory. Many communication systems, other than humans, are of course multimodal. These accept multiple modal local inputs and remote inputs over a network. A Windows desktop can have many possible active windows associated with different applications, it may not always be clear which window is active and accepts input. When a sender broadcasts or multicasts a request, it cannot predict who, if anyone, will respond to or fulfill a request. In social models, this is often used to find the first recipient who can help them, e.g. B. a call for help.

9.2.1.3 Too Many Messages Recipients may become overloaded with events that cannot be processed or stored in a timely manner as more UbiCom systems become online and available. For example, if a remote site allows remote senders to communicate synchronously anywhere and at any time, a single-task receiver, if it is already interacting locally, must decide whether to interrupt the local interaction to handle the remoting interaction. Policy-based management can be used to prioritize incoming events and handle events sequentially, e.g. for example, messages or a first-come, first-served policy can be processed. Buffers can be used to temporarily store received messages that cannot be processed immediately, to allow slow receivers to catch up during less busy periods, to support near real-time synchronous messages, and to support asynchronous messages. Receivers can also support flow control, i. h Send signals to senders asking them to slow down, or your messages may be discarded. Proxies that act as filters, aggregators, or digests can be used to offload event processing from sinks. Social media ratings and recommendations can be used to filter multiple events and only accept those with a threshold above a certain level.

9.2.2 Interaction through mediators The use of a mediator1 as a mediator between the interacting participants has several advantages. It supports greater autonomy between requesters and providers by adding a layer of indirection to allow connected peers to dynamically join and leave interactions while allowing the interaction to continue (hot swapping). You can simplify the number of connections that would be required to connect each participant to all other participants, instead of each participant being connected to a broker. It simplifies the interaction by acting as a proxy to hide the complexity of the interaction from one or more of the participants. Allows you to exert greater aggregate expertise to guide interactions when

Other general terms related to the use of third parties to act as intermediaries between two or more parties include brokers, service middleware, and broker agents.

variety of interactions

283

Brokers cater to multiple underwriters. Runners can be modified to improve performance, e.g. B. optimized for high performance in a router or gateway. Intermediaries also have disadvantages: performance may fail because another intermediate node is used for data transmission. It uses centralized management for interaction and is a potential bottleneck and single point of failure. Mediator masquerades can occur. It is easier for a third party to instigate a man-in-the-middle attack to fool one or both parties. Due to the perceived failures of relying on intermediaries, pure P2P systems have been proposed, but these have their own problems, such as: Therefore, hybrid designs are often used in practice to combine the advantages of mediated interaction and unmediated, p. B. Intermediaries are used for at least part of the interaction, e.g. B. for node detection only.

9.2.2.1 Access to shared communication resources Multiple interdependent senders often transmit messages over a common access or backbone network channel. For example: local and remote wireless communications; convergence of audio, video, data and mobile; multimodal input human-device interfaces. If there is no transmission control or cooperation, multiple transmissions may interfere with each other; B. when multiple transmitters use the same frequency independently to control multiple devices, eg. B. Wireless doorbells and remote-controlled toys: the so-called hidden node problem. A common management technique from a network modeling perspective is to use multiplex access control to divide the capacity of a communication channel into multiple logical channels, one for each message signal or data flow. In a reverse process, known as demultiplexing, the original signals can be extracted on the receiver side. Examples of multiplex type access control are CDMA, FDMA, TDMA. Note that much of the diversity of communication is often handled by a lower level network protocol service and hidden from applications, so applications are not aware of this diversity when their data is being transmitted. There are two basic interference avoidance schemes: collision avoidance schemes (next section) and collision detection schemes. Collision detection is usually followed by a scheme to stop the progress of the collision and then repeat the interaction in a way that avoids the collision, e.g. B. the CSMA/CD scheme. Collision detection is also commonly used at the application level in computer games that simulate the physical world to prevent two objects from occupying the same space, e.g. B. to prevent something from going through a wall. Collision detection is also used in control systems such as robots, where a controller attempts to recognize and avoid obstacles when performing controlled tasks such as palletizing in a partially deterministic physical world (Section 6.7).

9.2.2.2 Access to shared computing resources Interaction often requires the use of shared resources when system capacity is planned in such a way that a resource cannot be allocated to a single user.2 This may be because the resources are expensive to install and maintain. Instead, systems can be designed to handle concurrent resource access when multiple users try to access the same resources at the same time or access different resources that depend on each other, e.g. B. Multiple data transactions can attempt to read and write the same record in a relational database schema. There are multiple schemas that can be used to share access, such as:

An example of a system design in which a resource was dedicated to one user was the circuit-switched design used in earlier telecommunications systems, which used an end-to-end network connection per user (Section 11.2.1).

284

Smart system interaction

Multiple Access Network Control Schemes already mentioned (Section 9.2.2.1). Information access control schemes can also be used. For example, to prevent multiple transactions from simultaneously reading and writing the same records in an RDBMS information system (Section 3.2.2.1), concurrency control techniques based on the initial process to access a record block access to that record3 for additional users. A third technique is to use access control schemes based on the circulation of one token per resource between users and require users to first purchase the token before granting them access to the corresponding resource, e.g. a token ring network. Systems can use an MTOS to allow many applications to run on a single device (Section 3.4.3). This has the advantage of great user comfort. The device behaves like a single portal for multiple applications. But these applications can easily conflict in the use of computer system resources, e.g. For example, two different applications want to stream AV content to the hard drive, but only one application can write to the hard drive at a time. There are various MTOS process scheduling designs to support concurrent resource access to shared resources based on fairness, priority, and first come first served (Section 3.4.3).

9.2.2.3 Mediation between requesters and providers In practice, two interacting parties often use mediators or active autonomous intermediaries4 to facilitate their interaction. The use of intermediaries has several advantages (see also Section 9.2.2). It promotes a weak coupling between the initiating (first) party and the responding (second) party. One of the fundamental problems facing designers of open systems, e.g. The Internet is the problem of connection (Davis & Smith, 1983) - finding others who may have the information or skills you need. The use of mediators improves peer discovery and service discovery (Section 3.3.2). Peers can dynamically share their service preferences or service capabilities using a generic standardized service representation. Instead of having to request information from each peer, the functions of all registered peers can be accessed in one place, a third party at a known static address. There are two specific types of information used in the discovery process: capabilities and service preferences. A service capability5 or service description specifies what types of requests can be serviced by providers. Service preferences define the limitations of the requester in selecting a service provider. These restrictions can be private, public, or a combination of both (Figure 9.3). There are different types of mediators and ways of classifying them. While traditional distributed systems often define only one type of broker, the directory, Decker et al. (1997) define a classification6 for the types of agents considering which party knows the skills and preferences at the beginning of the service call. This scheme creates nine different types of mediators (Figure 9.3). A dashboard keeps track of requests: Requesters post their issues or requests;

The complication here is that multiple transactional application processes may want to access multiple records and then lock them. Deadlocks can occur because one transaction Y may lock some records A and B, but not others like C and D, while another transaction Z may lock records C and D, but not A and B. There are several schemes that resolve deadlocks. , including the use of a transaction manager to monitor, detect, and resolve conflicts, and the use of two-phase locks (Watson, 2006). 4 Active brokers can initiate communication. These are autonomous entities and participate in distributed control, while passive intermediaries (Section 9.2.2) are usually under centralized control. 5 Service capabilities are also known as service descriptions and service announcements. 6 The analysis of the mediators differs from the classification of Decker et al. (1997) on the terminology used to name some of the different types of mediators, how requirements are distinguished from preferences, and why a broader set of properties is considered. This last aspect was also suggested by Wong and Sycara (2000).

variety of interactions

285

Who knows Capability Provider

Provider and Mediator P

requester

Suppliers and Applicants and Intermediaries

PAG

Proxy M

Who knows wishes and preferences

herr

service proxy

shipping Q

Matchmaker, Directory P

PAG

M Applicant and Intermediary

METRO

herr

User proxy, recommender

real estate agents

PAG

Mesa

??? PAG

PAG

METRO

Applicant & Broker & Provider

METRO

herr

???

Monitor, Rules

Figure 9.3 Designs for mediators based on who (requester, mediator, or provider) knows what, i. h who knows capabilities (solid arrows), service requirements and/or preferences (dashed arrows). Who initiates the flow of skills or preferences is indicated by the dot

Vendors can then ask the Blackboard agent for requests that they can process. Examples of whiteboard systems include newsgroups, bulletin boards, and recommendation systems.7 A broker can protect the privacy of both the applicant and the provider. The broker understands both preferences and capabilities and routes both requests and responses accordingly; Neither the requester nor the provider need to deal directly with each other in a transaction. A Matchmaker8 stores capacity announcements, which can then be queried by applicants. Applicants can select and contact any provider. A sender requires requesters or vendors to make the request directly without the need for an intermediary, although an intermediary may be present and serve to expedite further interaction. A proxy acts on the client side, client proxy (Section 3.2.2.4), on behalf of requesters to facilitate access to providers. Proxies can also hide the inner workings of the requester and even anonymize the requester from the providers. Client proxies containing AI, e.g. For example, goal-oriented people can interact with users at a much higher level by acting as personal assistants (Maes, 1994), allowing people to delegate goals to them, e.g. B. set that room to be a movie instead of having to specify the exact actions you want them to take. Proxies can act on the provider's side. A wireless network access point and a web portal are examples of a provider-side proxy.

A recommendation system can be thought of as a sort of Blackboard broker that anonymizes service preferences and aggregates them into ratings that applicants can independently search for. 8 A matchmaker is sometimes called a directory service or yellow pages service.

286

Smart system interaction

The last type is a general agent that sees all messages, service requests, and service announcements. This type of mediator can act as a monitor, regulator, and arbitrator to resolve any perceived interaction problems. There are some interaction patterns, marked as "???" which are discussed as an exercise. Other mediator design issues include the following. Brokers can only be used when creating an interaction session, e.g. B. a requester consults a directory service to find a provider and then interacts directly with the provider, or can be used during the interaction session, e.g. B. a runner. Some types of intermediaries, e.g. Agents, such as intermediaries and proxies, can provide a degree of anonymity to applicants or providers by concealing the identity of one or both parties. Arbitrators can be designed to support a variety of different representations for capabilities and preferences based on semantics versus syntactics, different levels of detail and structures for capabilities and preferences, and to store instance types of a participant versus state of a participant. Agents can be designed to support different types of interactions, such as: B. vendors only (directories) or applicants only (bulletin boards), or both. Brokers can be designed to support different types of services, such as lookup, browsing, single-value lookups versus multi-value lookups (for example, capacity type, price, QoS, reputation, etc.), and varying degrees of persistence and anonymity for stored. data sources Requesters and providers can expect a certain level of fairness and impartiality from mediators, but this depends on the ownership and business model of the mediator.9 Whether mediators are truly owned by a closed group of providers, or act as Proxies of this, applicants should be aware of this and note that the services offered are limited to a specific group of providers. Even where the mediator claims to be independent and impartial, mediators may be influenced and motivated by business incentives, their business revenue model, the way information is made available to mediators, to generate multiple results from a search for one particular way, eg. , providers can take names for themselves so that they appear first in the alphabetical ordering of multiple provider results. Mediators can act as a trusted third party. When two parties are relatively unknown or do not trust each other, mediators can act as a neutral third party that they both trust to regulate any exchange between them. Mediators must not change the value of the information exchanged, i. h fidelity to the copy or transmission remains unchanged and must not exceed the mandate of the party they represent by demanding excessive liability from any party they represent. Instead of blindly trusting the mediator, participants can use authorization delegation along with institutional regulation when interacting with mediators.

9.2.3 Interaction through cooperative participants Cooperative interaction allows multiple systems to work together. Collaboration is characterized by two main characteristics: synchronizing activities (coordination) and acting together (cohesion or organizational interaction). Collaboration is easier to manage when: Homogeneously designed systems interact; there is a centralized control; Systems are commissioned to serve others (for example, when a service

Early electrical communication systems, such as the telegraph and telephone, were synchronized in time and used human intermediaries to perform the technical task of connecting the caller to the system. The first successful electromechanical telephone switch was invented in 1891 by a Kansas City undertaker named Strowger who was reportedly fed up with local telephone operators diverting their calls from potential customers to a competing undertaker. and he was determined to develop a system that could take the operator. out of the loop. He developed a switching system that used contact arms that rotated on axes to make contact with one of ten contacts that represented a digit of a telephone number in an arc. Therefore, due to the perceived bias of human operators, an automated telephone operator system was developed.

variety of interactions

287

Table 9.2 Advantages and disadvantages of cooperative systems Advantages of the cooperative model

Disadvantages of the cooperative model

Distributed problem solving: solve it faster because more parts are being worked on in parallel. Each part of a system has incomplete information or problem-solving abilities, which can then be combined in meaningful ways. Delegation: You

Ubiquitous Computing: Smart Devices, Environments, and Interactions - PDF Free Download (2023)